Hot backups

[Craig Ringer]

> Certainly one can just copy the entire mail spool... getting MOST of hte
> messages, and the .headers, .cache, .index, etc. Is this safe?

Personally, I wouldn't just copy the spool from beginning to end while 
cyrus was running - all your indexes etc will become out of sync if mail 
is delivered to a mailbox while it's being copied. *ick*. You also run 
the risk of a mangled mailboxes.db if it's being modified while it's 
being copied.

What I'm doing here is using LVM on linux to take a snapshot of the mail 
spool (actually all of /var). I make a plain-text copy of the 
mailboxes.db then take the snapshot. I mount the snapshot on a temporary 
mountpoint, add imap/, spool/imap/, /etc/imapd.conf, /etc/cyrus.conf and 
/usr/local/cyrus (where all the binaries etc live on my system) into a 
CPIO archive, gzip it, and scp that to another host. That host later 
writes the cyrus backup archive to tape.

This is automated and happens nightly. Additionally, both the most 
recent and second-most-recent backup files are stored live on the other 
server, so I can quickly recover from a failure of the primary cyrus 
server that doesn't affect anything else.

Note that my method does /not/ ensure that Cyrus isn't modifying the 
mail spools as I take the snapshot, nor does it ensure that mailboxes 
aren't changed between the conversion of the mailboxes.db to text and 
the snapshot. It's not perfect. For my site, however, it's quite 
sufficient - at worst I'll have an AWOL mailbox or two, a mailbox 
slightly out of sync with it's indexes, or a truncated message.

I'm also running into an issue where occasionally the snapshot fails - 
it's taken while reiserfs is in the middle of a transaction, and the 
journal can't be replayed on mount because the snapshot is read-only. 
This is more of an LVM issue than anything else. I get around it by 
syncing the disk before taking the snapshot, and by deleting the 
snapshot and trying again if it's not useable.

I'd love to know of a way to ask cyrus to defer message delivery briefly 
and make sure everything is in sync. That way I could signal cyrus to 
"sync up and pause", do the db conversions, snapshot the volume, and 
resume cyrus. I'd prefer to avoid killing and restarting the master 
during the normal backup process.

> What about the databases? Are they save to just copy? are they
> recoverable? Can they be rebuilt from the plain messages?

My understanding is that the mailbox contents can be rebuilt from the 
messages, though I seem to remember you'll lose all your flags like 
read, etc. The mailboxes.db _can not_ be reconstructed easily and must 
be backed up; the rest, I just don't know enough about. I understand 
that you can afford to lose the rest of the DBs.

> Isn't that
> what matters most, the messages?

The messages and and your mailboxes DB, yes. Though users will be /very/ 
unhappy if you lose their subscriptions and/or read status. Recovery 
without a copy of your mailboxes db  is, I understand, a total 
nightmare. I make a plaintext copy so that if I'm restoring the mail 
spools etc onto a host with a different berkley DB environment I can get 
up and running quickly and easily.

Anyway, I'm extremely far from an expert on this, having been running a 
Cyrus server for only a few months. If I'm wrong on some points, please 
let me know folks.

This area is one where I'm going to try to add some info to the wiki. 
There's some there, but it seems a little sparse and maybe I can help.

Craig Ringer