UC Davis Cyrus Incident September 2007

[Rob Mueller]

Thanks for your post, it's always interesting to hear other peoples stories.

> 1st STEP:  Perdition mail-proxies

> in a load-balanced pool and 2 can handle the load most days. Initially we

If you have a chance, definitely think about changing from perdition to 
nginx. There's slightly more work in that I think you'll have to write a 
custom daemon to do LDAP -> nginx http auth protocol mappings, but if you've 
got any perl experience that should be like a few dozen lines with 
Net::Server and Net::LDAP. With large numbers of IMAP connections, the 
improvements were considerable.

http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/

> for this in both the OS and the application.  There is some bottleneck for
> a resource that once it reaches a certain busy-ness level, everything 
> starts

> on our hardware such as Zones.  The theory being 2 or 3 Zones on a T2000
> with say 10K users each, would still let us accomodate the same number of
> users on the new hardware that we had originally targetted.

> The root problem seems to be an interaction between Solaris' concept of
> global memory consistency and the fact that Cyrus spawns many processes
> that all memory map (mmap) the same file.  Whenever any process updates
> any part of a memory mapped file, Solaris freezes all of the processes
> that have that file mmaped, updates their memory tables, and then

Ick, that will be nasty. This is going to particularly be a problem with a 
skiplist mailboxes.db. Basically every imapd will have an mmaped copy of the 
entire mailboxes.db in it's process map. Linux does not seem to have a 
problem with this at all (at a time in the past we had a 150M mailboxes.db 
file with 6000+ imapd processes and it worked fine). Clearly there's some 
low-level OS design issue that means that it's not as easy to share mmaped 
pages between processes in Solaris compared to Linux.

One option you have is that rather than creating separate "Zones" in the OS, 
you just create separate cyrus instances yourself. We do this at FastMail. 
Basically we've partitions all our storage into 300G units, and each 
partition has a completely separate config dir, cyrus.conf, imapd.conf, 
mailboxes.db, partition, etc and runs a separate cyrus master process. It 
required some automation (eg we have scripts that auto-generate the 
appropriate cyrus.conf, imapd.conf and init scripts), and it means every 
cyrus command run needs the -C parameter to tell which instance you're 
dealing with, but it means that each instance of cyrus is much smaller (eg 
much smaller mailboxes.db file, deliver.db file, etc) and there's less 
"global" contention points.

Actually the reason for breaking up into smaller units was for replication 
reasons, but despite that, I think it's been a good move in many respects. 
Of course if you have global shared folders or want a a murder environment, 
then that probably just makes things harder and each instance needs a 
"complete" mailboxes.db file I believe anyway, but we've decided that we 
don't need the one main thing murder offers which is global shared folders.

Rob