Today's pop quiz: replication

[Michael Menge]

Hi,


Quoting ellie timoney <ellie at fastmail.com>:

>>>> - a single cyrus instance may be the primary server for some users
>>>>   but a
> replica server for other users
>>> Are you sure about that?
>>
>> I'm
>  not sure about any of this.  But you make a good point: I thought I
>  could see a way that this was possible, but now I don't think I can.
>
> I think I can, again, though I guess not in a murder.
>
> But in a non-murder setup, I believe the only difference between a
> primary vs replica is "which one the user interacts with"?  Which, in a
> non-murder setup, one is presumably managing in some way external to
> cyrus (e.g. database, proxies, and some glue)...
>
> So it seems like it should be plausible to have two (or more) cyrus
> instances (hosted wherever), each running a sync_server plus a
> channel/sync_client for each of the others, and whatever happens on any
> replicates to the others.  You'd be trusting your outer, non-cyrus layer
> to make sure not to proxy any individual user to the wrong instance (at
> least, not while there's pending replication transactions for them).
> And I guess shared folders would be thorny.  But I don't (yet) see a
> reason from a replication standpoint why this wouldn't work.
>
One possible problem I am seeing is that after a split brain, or switch of
primary and replica for some users there might be some problems for renamed
folders, subsciptions and folder annotations. Cyrus is able to handle changes
on both sides in a mailbox  (new mail, deleded mail, changed mail) because
of modseq, but this is missing for mailbox metadata.



> So:
>
> - a single cyrus instance may* be the primary server for some users but
>   a replica server for other users
>
> [* with caveats and requiring custom tooling, and specifically not
> in a murder]

On the other hand working with multiple instances on one host is working fine
(even in an murder setup).

See http://www.spinics.net/lists/info-cyrus/msg16035.html

>
> ellie
>
> On Fri, Jul 24, 2015, at 10:01 AM, ellie timoney wrote:
>>
>>> Okay, I'll bite.  Here's what a bit of a sync_log looks like:
>>
>> Thanks!  So anything processing a sync_log (sync_client, squatter,
>> etc) needs to look at an actual copy of the user/mailbox in order to
>> determine its current state, and needs to look at both copies to work
>> out what to replicate between them.
>>
>>>> - a single cyrus instance may be the primary server for some users
>>>>   but a
> replica server for other users
>>> Are you sure about that?
>>
>> I'm not sure about any of this.  But you make a good point: I thought
>> I could see a way that this was possible, but now I don't think I can.
>>
>> Cheers,
>>
>> ellie
>>
>> On Thu, Jul 23, 2015, at 11:42 PM, Nic Bernstein wrote:
>>> On 07/23/2015 01:14 AM, ellie timoney wrote:
>>>>> Is the file format of the sync log defined anywhere? I assume it
>>>>> > correlates with a set of commands. (Not that this is important to
>>>>> > a user: it may as well be opaque, but it made me wonder!)
>>>> I'm a bit confused about this myself.  Each time I go digging
>>>> into the
> code my understanding flips back the opposite way.
>
> I think, either:
>
> * the sync log contains all the information needed to reproduce what's
>   happened (e.g. if a message has arrived, the sync log will contain the
>   message itself); OR
> * the sync log contains just enough to identify things that have changed
>   (e.g. if a message has arrived, the sync log contains a message id of
>   some sort), and the sync_client processing the log just uses the log
>   to discover which things to sync, but then uses the actual mailbox to
>   construct the changes to send to the replica.
>
> Either way I haven't seen any documentation on the sync log format.  I
> suspect it's either the raw sync protocol or some subset thereof?
>>>
>>> Okay, I'll bite.  Here's what a bit of a sync_log looks like:
>>>> MAILBOX user.newjersey
> MAILBOX user.support USER onlight USER nic USER admin USER randy MAILBOX
> user.randy.Trash USER lynn MAILBOX user.lynn.Trash
>>> It's basically a list of either users or mailboxes which have been
>     altered.  When sync_log is enabled, all of the daemons which might
>     alter a mailbox or user will write a line to this log each time they
>     do so.  That means the obvious suspects -- imapd, pop3d, timsieved,
>     lmtpd, etc. -- but also cyr_expire and friends (as in the
>     USER...MAILBOX couplets at the end of the sample).
>>>
>>>> - a single cyrus instance may be the primary server for some users
>>>>   but a
> replica server for other users
>>> Are you sure about that?  How does one specify the users for which
>     such an instance would play each role?  A single HOST may run
>     separate instances, which may perform these different roles, but I
>     cannot fathom how to configure a single instance to do both at once
>     for different user cohorts.
>>>
>>> This raises potential problems when one deploys replication within a
>     murder (Cyrus aggregation).  Only one server may claim ownership of
>     any given mailbox, via a mupdate call, so an instance which is a
>     replica MAY NOT push updates to mupdate master, or mayhem will
>     ensue.  Here's a commented section from /etc/cyrus.conf on a
>     replication master instance:
>>> ##
> # Master sends mailbox updates to mupdate. Replication client runs on
> # Master. Comment these 2 lines out on replicas
>         mupdatepush             cmd="/usr/lib/cyrus/bin/ctl_mboxlist -m"
>         syncclient              cmd="/usr/lib/cyrus/bin/sync_client -r"
>>> A nice daydream of mine envisions a world wherein mailboxes.db keeps
>     track of replica locations, as well, which would allow for the dual-
>     role operation Ellie describes.
>>>
>>> Cheers,    -nic
>>> --
> Nic Bernstein                             nic at onlight.com Onlight llc.
> www.onlight.com 219 N. Milwaukee St., Ste. 2A             v.
> 414.272.4477 Milwaukee, Wisconsin  53202               f. 414.290.0335
>>



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen