Recomendations for a 15000 Cyrus Mailboxes

[Andrew Morgan]

On Tue, 10 Apr 2007, Nestor A. Diaz wrote:

> Hello People,
>
> I am a proud cyrus user and fan, right know i need o implement a 15K users 
> cyrus mailstore, since i have been using cyrus for an average  400 users 
> mailstore, i will like to hear comments for setting up a cyrus for a 15K 
> users.
>
> I currently have a  ' postfix / postgresql / cyrus-sasl / cyrus ' production 
> server working for more than two years without problems, and i am going to 
> replicate the same configuracion, however since i am going to have a  huge 
> space  for mail  and people are going to use a lot of space, aprox. 50 MB per 
> user, i will like to hear some comments about what should be the best  (cyrus 
> / pop3 / imapd) configuration, i have read on the mail list some 
> configuration that fastmail.fm use, including pop3, imap proxies and multiple 
> instances of cyrus, but i like to see more comments about that.

I won't make too many recommendations.  Here is what we use at Oregon 
State University.

> These are the plans: (comments on every number will be apreciated)
>
>  1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB)

We have two 800GB RAID 5 partitions on an EMC Cx500 SAN (hardware RAID). 
Each partition is 7 10k 146GB drives.

>  2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ?

We used ext3 here for simplicity and reliability.  I don't have any 
experience with other filesystems, and this didn't seem like a good 
project to use as a testbed.  People get angry when mail doesn't work.

>  3. Which options to format the filesystem ? acording to the chosed
>     filesystem

Both partitions were formatted with the following commands:
   mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1
   tune2fs -c 0 -i 0 /dev/sdb1

Our mount options are "defaults,data=ordered,noatime".  You'll definately 
want "noatime" regardless of which filesystem you choose.

>  4. Which pop3 / imap proxy to use ?

We use Cyrus Murder with 2 backends, 2 frontends, and 1 Murder master (5 
separate servers).  The 2 frontends are load balanced using a ServerIronGT 
hardware load balancer.

>  5. Single instance or multiple instances of cyrus ? taking in mind
>     that there should be the option to recover a mailbox or some mail
>     of a mailbox without having to shut down the whole cyrus system.
>  6. Best way to perform backups ? LVM snapshots ? shutting down some
>     cyrus partitions ? RAID10 hot swap ?

We do online backups and restores all the time.  The entire Cyrus mail 
spool does not need to be consistent at a point a time, just an individual 
mailbox.  We use regular Legato filesystem backups, nothing special. 
Single mailbox restores are pretty simple too.  Just run reconstruct on 
the mailbox and update the quotas afterwards.

>  7. Any other suggestion will be welcome.

We use Postfix as our campus mail relay, which delivers mail to the Cyrus 
frontends using LMTP.  All spam filtering and blocking happens there, so 
our Cyrus servers just handle IMAP connections.

 	Andy