Replication documentation anywhere?

[Bron Gondwana]

On Sat, May 25, 2013, at 05:40 AM, Adam Tauno Williams wrote:
> 
> On Fri, 2013-05-24 at 09:41 +0100, Karl Pielorz wrote:
> > --On 24 May 2013 11:10 +1000 Bron Gondwana <brong at fastmail.fm> wrote:
> > > I'm afraid there isn't much :(  Feel free to ask questions about
> > > specific things you run in to, and we can use that as a basis to put
> > > together more detailed documentation.
> > Thanks for your reply (which I've read through now!) - if we have an 
> > existing mailstore (with ~2-3 million emails) and we setup a replica is 
> > 'sync_client' going to be able to bring the replica to the same state as 
> > the master? (given time, and not too many changes on the master?) - or are 
> > we better off using another method (such as FS snapshot -> replica).
> 
> Yes, a syncrepl will theoretically transfer all the contents of the
> server.  But it might be rather brutal.

On what do you base this opinion?

> Personally I would:
>  (1) rsync the message store from the master to the slave [this will
> bulk move tons of files,  mailboxes moved first could be quite out of
> date by the end.]
>  (2) rsync the message store from the master to the slave [this will
> only moved changed stuff, and give you a reasonably current snapshot of
> the mailstore.
>  (3) rsync the meta-data [/var/lib/imap] from the master to the slave.
>  (4) run reconstruct on the slave to make sure everything is consistent

Seriously?  Instead of using a protocol which is designed for
consistent transfer of an exact mailbox state you would rsync the
underlying files inconsistently, and then run reconstruct to bring it
back to something approximating consistency?  You'll certainly lose flags
on a few messages, not to mention how badly things will break if a mailbox
is created or deleted between rsyncing the meta and the data for that
same mailbox.

sync_client (at least in 2.4) is quite efficient in terms of IO.  It does
use a little more CPU because it's actually doing consistency checks on
everything, and maybe a little more IO on your meta files because it uses
those rather than the filesystem to determine which new messages to sync
every time - but then that's the same way it will be checking when it comes
to your later production syncs anyway.

>  (5) run "sync_client -v -r  -z" in screen and watch it for a bit,
> verify that messages get replicated as expected
>  (6) switch to running sync_client 'for real'
> 
> You can run "sync_client -z -u {user}" to sync a user, so if you have an
> easy to use list of all your users, or "-m {mailboxes}" if you generate
> a list of all your mailboxes.  This can help to 'push' the towards
> making sure the replica is up to date;  and running multiple
> sync_client's seems to work OK.

Yep - running multiple sync clients is fine.  If you sync the same
mailboxes/users in two different clients you will waste a bit of bandwidth
and IO, but it will still come to the same result - the second sync will be
a NOOP.

All the rsync stuff above though - it's extra complication and it actually 
means inconsistencies.  I don't even... no, I seriously don't take such a
crazy course of action into account when I'm thinking about the replication
protocol.  It would be like copying active table files out from under a
database engine and expecting it to cope.  Cyrus isn't designed for this at
all.  All sorts of stuff is transactional across multiple files.

The ONLY time I use rsync is when I'm going to be shutting down the cyrus
instance and doing a final rsync - in that case you get the consistency with
a giant "global lock".

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm