restore from cyrdump

[Andrew Morgan]

On Mon, 15 Dec 2014, Patrick Goetz wrote:

> On 12/10/2014 03:47 AM, Willy Offermans wrote:
>> I'm not sure what you mean with ``all the metadata'', but there are user
>> flags saved into the cyrdump file as well. I never performed the whole
>> cycle of dump and restore (probably nobody did so far), so I cannot tell
>> you that all metadata is available in the dump file. See my question above!
>>
>
>
> A while back I was working on an email server that (unbeknownst to me)
> was connected to a UPS, but with an external disk array that was plugged
> in to an outlet on the UPS that was not battery-backed.  This site had
> frequent power problems, so it turns out that power cycling a disk array
> while the server stays up is an awesome way to corrupt your entire file
> system.
>
> Since I didn't know what I was doing at the time, I restored
>
>   partition-default: /home/cyrus
>
> without also restoring
>
>   configdirectory: /var/lib/cyrus
>
> I was consequently confused when no mailboxes showed up, and had to then
> learn about and use reconstruct -r on each individual mailbox
> (cyrreconstruct on debian/Ubuntu) in order to reconstruct the
> /var/lib/cyrus/*.db files.
>
> I think the main database files you need are mailboxes.db and
> annotations.db (can someone confirm this?)
>
> This still leaves the question of how best to back up a cyrus mailstore.
>
> Bron mentioned that most people are using LVM snapshots.  I don't see
> how using btrfs/LVM/ZFS snapshots can save you from a race condition
> between when the cyrus user directory is updated and when mailboxes.db
> is updated.  The only way I would trust this is by doing this:
>
>    1. Stop cyrus
>    2. Snapshot
>    3. Restart cyrus
>
>
> cyrdump:  near as I can tell the only useful purpose this serves is to
> assemble all email messages into a single "mbox" file (can anyone
> confirm this)?
>
> ctl_mboxlist: this seems useful for making a human readable copy of the
> mailboxes.db file, but I'm not sure how this could be useful for
> disaster recovery, given the previously mentioned issue about keeping
> the mailboxes.db file synchronized with the contents of the user dir.
>
> So, given a simple mail server (i.e. no murder + replication), and when
> using a filesystem (e.g. ext4 or XFS) which doesn't do snapshots, it
> would appear that the only safe way to backup up a cyrus mailstore is to
> either using something like imapsync, or
>
>    1. Stop cyrus
>    2. tar cvf /some/safe/place/user.tar {default-partion}
>    3. tar cvf /some/safe/place/cyrusdb.tar {configdirectory}
>    4. Restart cyrus
>
> The way I've used imapsync in the past required copying mail folders per
> authenticated user account; i.e. something like
>
> imapsync --host1 my_host1 --authmech1 LOGIN --user1 my_user1 --password1
> xxxxx --host2 my_host2 --authmech2 PLAIN --user2 my_user2 --password2 xxxxx
>
> which in particular means knowing everyone's passwords.  This is
> entirely unworkable for larger sites, and I'm not sure if there is a
> trick for getting around this.

We are a "large" site.  We have 3 backend servers in a Murder cluster with 
about 25,000 mailboxes per backend.

Unless you are going to use Fastmail's fancy backup method that actually 
locks the mailbox in Cyrus, I don't think there is a way to take a 
perfectly consistent backup.

That said, we didn't care about achieving perfection, so we just perform a 
normal filesystem backup.  We use EMC Networker, but it isn't doing 
anything fancy.  It just walks the entire directory tree looking for 
changed files to backup.

Yeah, there is the potential for a race condition.  You really only need 
each mailbox to be consistent though, and the odds of a mailbox changing 
while it's being backed up are sufficiently low here.  In the event of a 
disaster (total loss of filesystem), we'll be restoring from our most 
recent backup, which will be anywhere from 0-24 hours old.  If we need to 
recover an individual mailbox, we can get everything using delayed delete. 
Either way, we'll be running reconstruct on the mailbox(es).

I see people talk a lot about fancy ways to back up Cyrus, but we've just 
never had a need.

 	Andy