LARGE single-system Cyrus installs?

[Bron Gondwana]

On Fri, Nov 09, 2007 at 01:28:05PM -0500, John Madden wrote:
> On Fri, 2007-11-09 at 19:10 +0100, Jure Pečar wrote:
> > I'm still on linux and was thinking a lot about trying out solaris 10,
> > but
> > stories like yours will make me think again about that ...
> 
> Agreed -- with the things I see from the Solaris (and zfs) and Sparc
> hardware in general, my money's still on Linux/LVM/Reiser/ext3.
> 
> 250,000 mailboxes, 1,000 concurrent users, 60 million emails, 500k
> deliveries/day.  For us, backups are the worst thing, followed by
> reiserfs's use of BLK, followed by the need to use a ton of disks to
> keep up with the i/o.

For us backups are hardly a blip on the radar :)  The joy of writing
your own custom backup system that knows more about Cyrus internals
than just about anything else.  It starts with some stat calls, and
if any of the cyrus.header, cyrus.index or cyrus.expunge files have
changed then it will lock them all then stream them all to the backup
server.

The backup server then parses them and decides (based on GUID) if
there are any data files it hasn't yet fetched.  If so, it fetches
them and checks the sha1 of the fetched file against the GUID.

The whole thing takes a couple of seconds per user and requires
less IO than even using direct IMAP calls would.

Now our big IO user is cyr_expire.  We run it once per week, and
it's a killer.  I'd be tempted to run it a lot more frequently if
it didn't have such a high baseline IO cost on top of the actual
message unlinks (though the unlinks are the real killer)

Bron ( and the BKL, *sigh*.  I just installed an external RAID
       unit with 8x1TB drives in it for data.  That 6GB/300Mb == 20
       data partitions plus 20 meta partitions to go with it.  That's
       a lot of BKL! )