LARGE single-system Cyrus installs?

[Bron Gondwana]

On Tue, Nov 13, 2007 at 10:24:22AM +0000, David Carter wrote:
> On Sun, 11 Nov 2007, Bron Gondwana wrote:
> 
> >> 250,000 mailboxes, 1,000 concurrent users, 60 million emails, 500k 
> >> deliveries/day.  For us, backups are the worst thing, followed by 
> >> reiserfs's use of BLK, followed by the need to use a ton of disks to 
> >> keep up with the i/o.
> >
> > For us backups are hardly a blip on the radar :)  The joy of writing 
> > your own custom backup system that knows more about Cyrus internals than 
> > just about anything else.  It starts with some stat calls, and if any of 
> > the cyrus.header, cyrus.index or cyrus.expunge files have changed then 
> > it will lock them all then stream them all to the backup server.
> 
> Cyrus is pretty ideal for fast incremental updates to a backup system: 
> hence replication. You shouldn't need to lock anything with delayed 
> expunge, delayed delete and fast rename in place.

If you're planning to lift a consistent copy of a .index file, you need
to lock it for the duration of reading it (read lock at least).

Yeah - replication is one way to do it.  We happen to read from the
masters at the moment, but it would be pretty trivial to switch to
using the replicas (change a $Store->MasterSlot() to
$Store->ReplicaSlot() at one place in the code in fact) if we wanted to.

But since I would like a consistent snapshot of the mailbox state,
I lock the cyrus.header and then the cyrus.index and then (if it's
there) the cyrus.expunge.  That means no sneaky process could (for
example) delete the mailbox and create another one with the same
name while I was busy downloading the last file - giving me totally
bogus data.  This is particularly important because I store things
by mailbox uniqueid rather than imap path (with pointers from the
imap path of course) so that a folder rename turns into a symlink
delete (well, replacement with one having an empty target anyway) 
and a symlink create in the tar file.

Bron ( and right now I'm running the process to finish the upgrade
       from MD5 based to SHA1 based internal identifiers in the
       backup system, since all our indexes are upgraded )