<div dir="ltr">Hi,<div><br></div><div>Any news about <span style="font-size:12.727272033691406px;font-family:arial,sans-serif"> </span><span class="" style="font-size:12.727272033691406px;font-family:arial,sans-serif">multi</span><span style="font-size:12.727272033691406px;font-family:arial,sans-serif">-</span><span class="" style="font-size:12.727272033691406px;font-family:arial,sans-serif">master</span><span style="font-size:12.727272033691406px;font-family:arial,sans-serif"> replication?</span><br>
</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/5/24 <span dir="ltr"><<a href="mailto:cyrus-devel-request@lists.andrew.cmu.edu" target="_blank">cyrus-devel-request@lists.andrew.cmu.edu</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Cyrus-devel mailing list submissions to<br>
<a href="mailto:cyrus-devel@lists.andrew.cmu.edu">cyrus-devel@lists.andrew.cmu.edu</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.andrew.cmu.edu/mailman/listinfo/cyrus-devel" target="_blank">https://lists.andrew.cmu.edu/mailman/listinfo/cyrus-devel</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:cyrus-devel-request@lists.andrew.cmu.edu">cyrus-devel-request@lists.andrew.cmu.edu</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:cyrus-devel-owner@lists.andrew.cmu.edu">cyrus-devel-owner@lists.andrew.cmu.edu</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Cyrus-devel digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Replication documentation anywhere? (Bron Gondwana)<br>
2. Re: Replication documentation anywhere? (Karl Pielorz)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 24 May 2013 11:10:16 +1000<br>
From: Bron Gondwana <<a href="mailto:brong@fastmail.fm">brong@fastmail.fm</a>><br>
Subject: Re: Replication documentation anywhere?<br>
To: <a href="mailto:cyrus-devel@lists.andrew.cmu.edu">cyrus-devel@lists.andrew.cmu.edu</a><br>
Message-ID:<br>
<<a href="mailto:1369357816.25848.140661234996321.590D3BC2@webmail.messagingengine.com">1369357816.25848.140661234996321.590D3BC2@webmail.messagingengine.com</a>><br>
<br>
Content-Type: text/plain<br>
<br>
On Fri, May 24, 2013, at 01:34 AM, Karl Pielorz wrote:<br>
> We're running cyrus-imapd-2.4.17 on FreeBSD. I've been looking at the<br>
> replication built into Cyrus, but can't find much (if any)<br>
> documentation on it.<br>
><br>
> e.g. The shipped 'install-replication.html' file ends at:<br>
><br>
> " ... You can also run cyr_synclog(8) instead, which will insert the<br>
> record into the rolling replication log.<br>
><br>
> Failover "<br>
><br>
> And that's it. Is there anywhere I can find more info on replication /<br>
> failover / setup, a 'howto' - or anything?<br>
<br>
I'm afraid there isn't much :( Feel free to ask questions about<br>
specific things you run in to, and we can use that as a basis to put<br>
together more detailed documentation.<br>
<br>
Failover is kind of messy at the moment, because it's so site-dependent<br>
how you want to manage your failover. Our process at FastMail looks<br>
like this:<br>
<br>
1) update database to mark the server as "moving" so new connections get<br>
paused at nginx/web server level and then wait 2 seconds for the<br>
config to be updated.<br>
2) send a signal to the 'master' process to shut the server down.<br>
3) wait for up to 10 seconds while grepping the process list every<br>
second for ongoing processes related to the same instance (we use -C<br>
$imapd_conf because we run many instances of Cyrus per server)<br>
4) if the processes aren't dead after 10 seconds, kill them<br>
individually.<br>
5) if THAT fails, kill -9. (yeah, I know - evil!)<br>
6) check the $confdir/sync directory for log files, and run them with<br>
sync_client to ensure all replication is up to date.<br>
<br>
If anything before this failed, we bring this master back up and report<br>
that the failover didn't succeed.<br>
<br>
7) shut down the replica<br>
8) restart this side with the replica configuration<br>
9) restart the other side with the master configuration<br>
<br>
At the moment, we still move a master IP address to the instance which<br>
is running as master, meaning clients can reconnect to the same IP<br>
address after the failover. This is on its way out - we're now at the<br>
point where almost everything can read configuration from our<br>
"fmstatus.json" file which is updated every second on every host, so<br>
they know where the master is actually located.<br>
<br>
Obviously, a ton of this is really site-specific to us.<br>
<br>
Soon (yes, soon!) we will be shifting to a full multi-master setup,<br>
where failover is as simple as pointing clients to the other end of<br>
the replication pair, and killing off existing connections so they<br>
reconnect (with some sync_log checking and force-running), which should<br>
shave quite a few seconds off the sync time and mean that long running<br>
squatter jobs and other things don't get nuked off at the same time.<br>
<br>
But yeah, it's not quite a turn-key system :(<br>
<br>
Bron.<br>
<br>
--<br>
Bron Gondwana<br>
<a href="mailto:brong@fastmail.fm">brong@fastmail.fm</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 24 May 2013 09:41:17 +0100<br>
From: Karl Pielorz <<a href="mailto:kpielorz_lst@tdx.co.uk">kpielorz_lst@tdx.co.uk</a>><br>
Subject: Re: Replication documentation anywhere?<br>
To: Bron Gondwana <<a href="mailto:brong@fastmail.fm">brong@fastmail.fm</a>>,<br>
<a href="mailto:cyrus-devel@lists.andrew.cmu.edu">cyrus-devel@lists.andrew.cmu.edu</a><br>
Message-ID: <<a href="mailto:B2787CE797592C5F029D55A6@Mail-PC.tdx.co.uk">B2787CE797592C5F029D55A6@Mail-PC.tdx.co.uk</a>><br>
Content-Type: text/plain; charset=us-ascii; format=flowed<br>
<br>
<br>
--On 24 May 2013 11:10 +1000 Bron Gondwana <<a href="mailto:brong@fastmail.fm">brong@fastmail.fm</a>> wrote:<br>
<br>
> I'm afraid there isn't much :( Feel free to ask questions about<br>
> specific things you run in to, and we can use that as a basis to put<br>
> together more detailed documentation.<br>
<br>
Hi,<br>
<br>
Thanks for your reply (which I've read through now!) - if we have an<br>
existing mailstore (with ~2-3 million emails) and we setup a replica is<br>
'sync_client' going to be able to bring the replica to the same state as<br>
the master? (given time, and not too many changes on the master?) - or are<br>
we better off using another method (such as FS snapshot -> replica).<br>
<br>
Presumably the replication doesn't do anything for users (we use sasl for<br>
auth, i.e. saslpasswd2 to create the users) - if we intend to have users<br>
'hitting' the replica (when the master as failed) presumably we'd need to<br>
create those users / passwords on the replica as well?<br>
<br>
In a nutshell - when we have the master / replica setup, if we 'switch' to<br>
having users hit the replica (because the master has failed) - it appears<br>
we can simply 'switch' the replication configs over, and then have what was<br>
the master 're-sync' with the new master?<br>
<br>
I guess what I'm trying to figure out is if the replication / 'sync_client'<br>
is the 'magic' solution is appears to be - i.e. given enough bandwidth,<br>
time / 'grunt' & low enough volume of changes on the master - it'll drag an<br>
empty mailstore up to be an identical replica, or re-sync a once failed<br>
master to be a new replica.<br>
<br>
Thanks again for your time,<br>
<br>
-Karl<br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Cyrus-devel mailing list<br>
<a href="mailto:Cyrus-devel@lists.andrew.cmu.edu">Cyrus-devel@lists.andrew.cmu.edu</a><br>
<a href="https://lists.andrew.cmu.edu/mailman/listinfo/cyrus-devel" target="_blank">https://lists.andrew.cmu.edu/mailman/listinfo/cyrus-devel</a><br>
<br>
<br>
End of Cyrus-devel Digest, Vol 92, Issue 8<br>
******************************************<br>
</blockquote></div><br></div>