sync_client behaviour improvement planning

[Bron Gondwana]

On Mon, 26 Feb 2018, at 10:37, Sebastian Hagedorn wrote:
> Hi,
> 
> we're only getting started with replication, so I can't contribute any> useful comments, but your post raised two questions for me:
> 
> This only affects the "new", post-3.0 replication protocol,
> right? We're> planning a migration from 2.4 to 3.0 using replication, and I'd
> like to be> aware of all potential issues we might encounter.

A migration will be totally fine regardless.

I realise now the danger of posting about a really rare edge case to the
dev list in a way that implies that it's anything anyone is likely to
hit in regular usage.  It surfaced something like 3 times over tens of
thousands of mailboxes and a full disk event causing some weirdness
around retries in sync.  Cyrus handles full disk quite well these days
(after an event in 2014 where we fixed a bunch of bugs!) but it does
make some things odd.  In particular, it can cause replication to fail
for a mailbox due to disk full, but then later mailboxes in the same
list get processed OK because there was space for that operation.
In summary, don't let your disks get full!

>> Hey - here's me posting something to the public list instead of
>> internal>> FastMail slack.  We've been really bad at making our random
>> ruminations>> public, sorry.
>> Tomorrow (Tues 27th) 2pm Melbourne time, I'm going to be meeting with>> ellie and maybe Partha in the Melbourne office with a
>> whiteboard and a>> screen to flesh out some ideas for things we can do to fix
>> some of the>> issues that came up after a recent machine failure event at FastMail.> 
> <snip>
> 
>> b) there are over 1000 mailboxes, and the log file got
>>    deduplicated and>>   then run in sets, and we had this:
>> sync_log MAILBOX Z
>> sync_log MAILBOX B
>> 
>> (for a rename of Z to B)
> 
> Where and when does deduplication happen?

Deduplication in this case is sync_action_list_add(mailbox_list,
args[1]) in sync_client.c:do_sync().
If the same item is logged twice in the same log file, then it's only
added once to the list of mailboxes later passed to do_mailboxes.
Bron.


--
  Bron Gondwana, CEO, FastMail Pty Ltd
  brong at fastmailteam.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20180226/e26d8c47/attachment.html>