Backup strategy for large mailbox stores

[Gavin McCullagh]

Hi,

I'm a relative newbie with cyrus, but I'm interested in this discussion...

On Mon, 15 Feb 2010, ram wrote:

> We have cyrus servers deployed at many places where clients have varying
> mail storage.
> We have been taking backups to help in situations of  human errors
> ( where you get complaints like ..oops, I accidentaly deleted all my
> mails!! )  and in case of hardware failures
> 
> Things have been working fine but off late we find that emailusage has
> grown and so our backups take too long to complete .. we use dar to take
> differential backups and take backups everynight. and transfer the
> backup files to a remote server. 

Have you identified the bottleneck?  Is it disk access on the mail server
itself, bandwidth to your remote server, something else?

> If the backup is still running in the morning people notice a
> considerable degradation of the server performance

Is this a recent linux server?  In principal, you could use ionice to class
your dar process "idle" which should mean that users will get a better
share of disk access.  However, that will also mean your backup takes even
longer.  Probably not ideal.

> Is there a better strategy , probably within the cyrus framework , to
> take backups efficiently 

I've wondered about the best means of backup myself.  We've been doing
something similar using rsync to sync the mail spools and other associated
data to a remote server.  This works, but I'm slightly worried that we
continue delivering mail throughout the process.  So our mail spool is
changing as we back it up.  I've considered the possibility of stopping all
daemons, taking an LVM snapshot, restarting and backing up the snapshot.
That way you get a consistent spool where everything was backed up at the
same moment.  On the other hand, it appears that you can generally
reconstruct mailboxes, so perhaps I just don't need to worry about that.
I'd prefer the cosy feeling of knowing the data is in a consistent state
though.

If you simply can't run an incremental or differential backup in the
"quiet" time, perhaps it would make more sense to do rolling replication to
another server.  Then, your backup can stop the replication temporarily,
backup the replica and start the replication back up -- leaving the live
server alone.  I imagine this does add load to the main server, but
distributes it over the whole day.

http://cyrusimap.web.cmu.edu/imapd/install-replication.html

Gavin