Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

On Fri, Nov 8, 2019, at 1:35 PM, Deborah Pickett wrote:
> I didn't know if copying 
> the filesystem of a (paused) Cyrus replica was a supported way of 
> backing up, but now I do.

Yeah, as long as there are no cyrus processes running, the database/index files can just be copied about and won't be corrupted along the way.  This kind of backup is useful for a full system restore, but the "help I deleted an important email and then emptied my trash and then expunged, and now I need it back" type of recoveries... I guess you copy the message file back to the partition, reconstruct, and it comes back as a new message (unread, no flags, etc)?

You would need to be careful of the window between delivery of a message, replication to the replica, and deletion of the message (and replication of the deletion), to ensure you get a backup of the state where the message existed.  I *think* delayed_delete and a long cyr_expire -D time takes care of this, but I'm not certain, so please test it before you rely on it.  Also (maybe obviously) the need to keep point in time snapshots for as long as your recovery policy dictates, and not just delete stuff from backup as soon as its deleted from source.

I'm not sure what others are doing in this space really.  There's a few threads on the list archive about various backup strategies, but my focus has mainly been the backupd-based system.

>  Is there a list of which database and index 
> files I need to copy apart from the files inside the partition structure?

This kind of covers it, I think:
https://www.cyrusimap.org/imap/reference/admin/locations/configuration-state.html

It would be quite useful to have a "this is what you need to back up" document, but at the moment there's a certain amount of reading between the lines of adjacent documentation :(

> > This setting might be helpful:
> 
> Thanks, I saw that setting but didn't really think through how it would 
> help me.  I'll experiment with it and report back.

That would be great, thanks!

Cheers,

ellie