cyrus replication over a WAN

[Bron Gondwana]

On Thu, Dec 10, 2009 at 12:37:51AM -0800, Jon . wrote:
> In the past I found that if Cyrus is restarted on the replica, the
> sync_client on the master server fails. I also saw instances of sync_client
> failing on the master if the replica isn't available. The lower levels
> pertaining to why I don't know (or remember), but I can recreate this and
> post the exact error that is logged if any one on this list is interested.

Yes, it sure does.  Hence the '-o' (try to connect only once) flag that
doesn't cause the master to fail to start if the replica is away!
 
> Here's a bash script executed by the cron daemon on a pair of Cyrus servers
> every minute for watch-dogging sync_client.

[snip]

Interesting, but pretty incomplete.  It doesn't deal with failed log files,
so replication won't be 100%.

> The script checks for the conditions that the server is set as the master
> (which it is set to if Cyrus is started) and that the sync_client is not
> running. If so it runs the sync_client again.

I'll attach our script.  It has to deal with the fact that there are multiple
sync_client and sync_server processes on every machine because we run multiple
instaces of Cyrus all over the place, so it matches on config file names
(we pass them to every process as a command line option, and it appears in
the ps output)  We run it from cron every 10 minutes on every cyrus machine.

(coming back to edit: I've just read through the code again, and I'm not
embarassed to post it ;)  It does depend very heavily on our Perl
infrastructure code - so it's not very portable to anywhere else!  But
the ideas are sound and pretty easy to understand if you know a bit of
Perl!)
 
> You can also use this script to enable bi-directional IMAP replication. Why
> you would want to do such a thing (with a two-server master/replica
> configuration) is if you have SMTP daemons running on both of your Cyrus
> servers and want the "replica"/"passive" (IMAP-side at least) server to be
> able to push mail accepted by it from the outside world onto the master
> Cyrus server. This is probably better (and faster for the user) than simply
> dropping packets from outside SMTP servers (which is what I've seen on
> two-server master/replica Cyrus servers out there, fencing was done through
> setting ports as "filtered'). Because a user's IMAP activity on the master
> server shouldn't conflict with incoming mail, in theory bi-directional Cyrus
> replication for incoming messages from the SMTP daemon is not only optimal
> but also possible without creating a split-brain scenario. I plan to test
> this myself when I have time.

Ouch.  User uploads (or copies) a message into their Inbox as the same time
as a message is delievered at the other end.  Cyrus happily overwrites the
copy on the "replica" end (whichever copy of sync_client happens to run first)

I've got some code to make that better - half written in fact.  I already have
code to bail and leave the situation unresolved, which is better than
overwriting, but it still nukes "future" messages, which is just as bad.
 
> This was all done with Cyrus 2.3.7 on CentOS 5.3 and 5.4 in each case. Your
> mileage may vary and if any of the behavior described above is expected to
> be different with new versions of Cyrus please share.

No, not really.  2.3.7 is OK, though it still has some modseq related issues
and a few skiplist bugs.  

I really, really wouldn't do what you're talking about doing though - make
the MTA on your replica go find the master end and LMTP to it.  The network
traffic will be twice as much (since it will replicate back again), but the
locking and UID creation will be guaranteed to work.

Regards,

Bron.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: monitorsync.pl
Type: text/x-perl
Size: 3745 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091210/6152e44b/attachment.bin