choosing a file system

[Rob Mueller]

> Running multiple cyrus instances with different dbs ? How do we do that.
> I have seen the ultimate io-contention point is the mailboxes.db file.
> And that has to be single.
> Do you mean dividing the users to different cyrus instances. That is a
> maintenance issue IMHO.

As Bron said, yes it is, but if you have more than 1 machines worth of users 
anyway, you have maintenance issues anyway. So rather than just one instance 
per machine, we run multiple instances per machine. The only issue it really 
introduces is that folder sharing between arbitrary users isn't possible 
(unless you used murder to join all the instances together again, but we 
don't), only users within an instance can share.

> I had the feeling whatever optimizations done at the FS level would give
> us a max of 5-10% benefit.
> We migrated from ext3 to reiserfs  on our cyrus servers with 30k
> mailboxes. I am not sure I saw a great benefit in terms of the iowait.
> At peak times I always see a iowait of 40-60%

To be honest, that's not what we saw in our ext3 <-> reiserfs tests.

What mount options are you using? Are you using the mount options I 
mentioned?

noatime,nodiratime,notail,data=ordered

> But the new Solid-State-Disks seem very promising. They are claimed to
> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times
> that should make all these optimizations unnecessary.
> As my boss used to tell me ... Good hardware always compensates for
> not-so-good software.

What we've found is that the meta-data (eg mailbox.db, seen db's, quota 
files, cyrus.* files) use WAY more IO than the email data, but only use 
1/20th the space.

By separating the meta data onto RAID1 10k/15k RPM drives, and the email 
data onto RAID5/6 7.2k RPM drives, you can get a good balance of 
space/speed.

Rob