Cyrus Replication (example) [was Re: restore from cyrdump]

Nic Bernstein nic at onlight.com
Thu Dec 18 10:59:25 EST 2014


Folks,
Following Willy Offermans' recent response to this thread, I've decided 
to jump in, but wish to quote a bit more of Patrick's original message 
(below), hence the out-of-sequence reply.

I'd like to address Patrick's issues in pretty much the order they're 
presented here.  Firstly, however, I would like to echo some confusion 
as to the purpose of cyrdump.  It seems to have been around since at 
least 2.2.3, but documentation has appeared only fleetingly, and says 
little about the program:

    NAME
            cyrdump - dump mailboxes to stdout

    SYNOPSIS
            cyrdump [-C <configfile>] [-v] [mboxpattern ...]

    DESCRIPTION
            A tool for dumping IMAP mailboxes on a server.

            -C <configfile>
                   Specify an alternate configuration file ( is used by default)

            -v     Increase program verbosity.

Not much help, that.

As for the utility of filesystem snapshots, be they LVM2, ZFS, XFS, 
etc., this of course extends only so far as the stability of the 
underlying data.  If your data is in flux at the time of the snapshot 
(unwritten buffers, etc.) then you'll only snapshot an incoherent, or 
"crash-consistent", state.  Thus it's important to consider the relative 
cost and consequence of taking the extra steps to quiesce the 
application prior to the snapshot.

We perform ZFS snapshots very frequently, without application 
quiescence, under the belief that even crash-consistent data is better 
than no data, in a DR scenario, and we replicate this data across 
multiple machines and sites, to further improve any potential DR 
outcome.  However, we also make "application-consistent" backups, which 
require application quiescence, as Patrick has described.  His initial 
suggestion -- stop cyrus, snapshot, restart cyrus -- is reasonable, but 
we feel that the later suggestion -- stop cyrus, tar up data, start 
cyrus -- is not.  It takes data offline for too long.  That's why the 
snapshot capability is necessary in any truly suitable server.

If you, like us, maintain your "config" and spool (and potentially meta) 
data on separate filesystems, you must remember to snapshot them all.  
Also, if you're going to stop and start cyrus like this, it pays to 
remove heavyweight processes (i.e. delprune, checkpoint, sqautter) from 
the START section of cyrus.conf, to prevent extensive delays at restart, 
and let them be performed periodically (in EVENTS) instead.

I will say that the ability to quiesce the application without halting 
it would be most desirable.  Most databases have supported this sort of 
thing for ages, and it would be great if one could send a signal to 
Cyrus to achieve the same result.  One wonders, however, if the model of 
master process (cyrmaster for Debian-based distros) with potentially 
hundreds or thousands of child daemons, would lend itself to this.

But snapshots, whatever their consistency or coherence, will always be 
less useful for DR than a replica, and in Cyrus this is not only 
available, but not really that hard to get up and running.  The 
differences between the imapd.conf files on a master and replica can be 
this simple, or simpler even, as this example is from a murder (from a 
running pair):

    --- imapd-master.conf	2014-11-24 13:37:45.752639666 -0600
    +++ imapd-replica.conf	2014-11-24 13:31:58.702611566 -0600
    @@ -1,22 +1,14 @@
      ##
    -# These configuration parameters are for the master server
    -# in a replication set
    -servername: master.example.com
    +# These configuration parameters are for the replica server in a
    +# replication cluster
    +servername: replica.example.com
      
    -##
    -# Auth credentials
    -mupdate_server: mupdate.example.com
    -mupdate_username: mupdate
    -mupdate_authname: mupdate
    -mupdate_password: <password>
    -
    -##
    -# Replication support
    -# This is how the BACKEND for this host is defined
    -sync_host: replica.example.com
    -sync_authname: mailproxy
    -sync_password: <password>
    -sync_compress: true
    -sync_log: true
    -sync_repeat_interval: 5
    -sync_shutdown_file: /var/run/cyrus/sync_stop
    +## Auth credentials
    +# The credentials below must match the account listed in lmtp_admins
    +# on the backend servers.
    +proxy_authname: mailproxy
    +proxy_password: <password>
    +force_sasl_client_mech: PLAIN
    +mupdate_mechs: PLAIN
    +mailbox_mechs: PLAIN
    +serverlist: master

The changes in cyrus.conf are similarly brief:

    --- master/cyrus.conf	2014-11-24 14:01:26.832602819 -0600
    +++ replica/cyrus.conf	2014-11-24 14:01:42.212637226 -0600
    @@ -15,8 +15,8 @@
      	# Master sends mailbox updates to mupdate.
      	# Replication client runs on Master.
      	# Comment these 2 lines out on replicas
    -	#mupdatepush		cmd="/usr/sbin/cyrus ctl_mboxlist -m"
    -	#syncclient		cmd="/usr/lib/cyrus/bin/sync_client -r"
    +	mupdatepush		cmd="/usr/sbin/cyrus ctl_mboxlist -m"
    +	syncclient		cmd="/usr/lib/cyrus/bin/sync_client -r"
      
      	# this is recommended if using duplicate delivery suppression
      	#delprune	cmd="/usr/sbin/cyrus expire -E 3 -D 60 -X 60" at=0400
    @@ -55,7 +55,7 @@
      	##
      	# Synchronization for remote replication.
      	# Comment this out on Master, uncomment on replicas
    -	syncserver       cmd="/usr/lib/cyrus/bin/sync_server" listen="csync"
    +	#syncserver       cmd="/usr/lib/cyrus/bin/sync_server" listen="csync"
      
      	# Experimental httpd for caldav
      	httpd		cmd="httpd" listen=8080 prefork=1 maxchild=20

It's just not that hard to implement, and for the security and safety it 
provides is well worth it.

Note that if you're using a murder, the server name must be preserved 
between the two hosts, in the event of a fail-over, and only the master 
should communicate with the mupdate server (as shown in my example).  If 
you're not using a murder, then most of these differences go away (all 
the mupdate stuff).  Of course, you'll need to switch DNS or have some 
other way of dealing with fail-over on layer 3.

We put all common configuration into /etc/imapd.conf, and then use the 
@include directive to include the appropriate file fragment, either 
imapd-master.conf or imapd-replica.conf.  For the cyrus.conf, it's just 
a couple of line edits -- commenting and uncommenting -- so we do that 
by hand.  You've got to intervene to update the DNS, in any event, so 
this much more is trivial.

Lastly, as to the use of imapsync to achieve user, mailbox or server 
replication, and the authentication requirements thereof, I would 
suggest reading the README file for that application, which includes this:

     You may authenticate as one user (typically an admin user), but be
     authorized as someone else, which means you don't need to know every
     user's personal password. Specify --authuser1 "adminuser" to enable this
     on host1.

So your command line is much like Patrick's example, but with '--user1 
<user> --authuser1 <proxyuser> --user2 <user>...'
Of course you must create a proxy user, and Cyrus supports this with the 
'proxyserver' directive in imapd.conf (man imapd.conf for details), 
i.e.: 'proxyservers:    proxyuser'.

However, I must be honest and point out that if you're going to go to 
the trouble of figuring out how to use imapsync (and possibly pay for 
it, to boot) you may as well just set up a replica.  As I've shown, 
above, it's just not that hard.

Cheers,
     -nic

On 12/15/2014 05:01 PM, Patrick Goetz wrote:
> This still leaves the question of how best to back up a cyrus mailstore.
>
> Bron mentioned that most people are using LVM snapshots.  I don't see
> how using btrfs/LVM/ZFS snapshots can save you from a race condition
> between when the cyrus user directory is updated and when mailboxes.db
> is updated.  The only way I would trust this is by doing this:
>
>      1. Stop cyrus
>      2. Snapshot
>      3. Restart cyrus
>
>
> cyrdump:  near as I can tell the only useful purpose this serves is to
> assemble all email messages into a single "mbox" file (can anyone
> confirm this)?
>
> ctl_mboxlist: this seems useful for making a human readable copy of the
> mailboxes.db file, but I'm not sure how this could be useful for
> disaster recovery, given the previously mentioned issue about keeping
> the mailboxes.db file synchronized with the contents of the user dir.
>
> So, given a simple mail server (i.e. no murder + replication), and when
> using a filesystem (e.g. ext4 or XFS) which doesn't do snapshots, it
> would appear that the only safe way to backup up a cyrus mailstore is to
> either using something like imapsync, or
>
>      1. Stop cyrus
>      2. tar cvf /some/safe/place/user.tar {default-partion}
>      3. tar cvf /some/safe/place/cyrusdb.tar {configdirectory}
>      4. Restart cyrus
>
> The way I've used imapsync in the past required copying mail folders per
> authenticated user account; i.e. something like
>
> imapsync --host1 my_host1 --authmech1 LOGIN --user1 my_user1 --password1
> xxxxx --host2 my_host2 --authmech2 PLAIN --user2 my_user2 --password2 xxxxx
>
> which in particular means knowing everyone's passwords.  This is
> entirely unworkable for larger sites, and I'm not sure if there is a
> trick for getting around this.

-- 
Nic Bernstein                             nic at onlight.com
Onlight, Inc.                             www.onlight.com
219 N. Milwaukee St., Suite 2a            v. 414.272.4477
Milwaukee, Wisconsin  53202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20141218/8560752b/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nic.vcf
Type: text/x-vcard
Size: 271 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20141218/8560752b/attachment-0001.vcf 


More information about the Info-cyrus mailing list