Cyrus IMAP running in linux: recommended file system

[Bron Gondwana]

On Thu, Dec 16, 2010 at 06:09:58PM -0800, David Lang wrote:
> On Thu, 16 Dec 2010, Lucas Zinato Carraro wrote:
> 
> > I know that this question is controversy and there is no exact answer.
> >
> > But  currently what is the best file system in  Linux to handle
> > thousands of small files?
> > I have mailboxes with 4Gb. And messages with ~4Kb.
> > Ext4 ?
> 
> depending on who you ask you will get
> 
> ext4
> btrfs
> XFS
> 
> my personal opinion is that ext4 is still new enought that I don't trust it 
> fully (they are still finding too many problems and having to scramble to fix 
> them)

Disagree - it's been quite stable for about a year now, and usable for at
least two.  The stuff they're finding is skanky little edge cases that
we haven't run in to at all, so I'm happy.

We're running about 50% on ext4.  About 1% on btrfs (a testing store, no real
users) 
 
> this makes btrfs much too new.

Agree.

> so I opt for XFS

Performance with zillions of small files isn't XFS's strong suit.

> you want to avoid ext2/ext3 as they do very poorly with large numbers of files 
> in one directory.

Definitely agree.  ext3 was unusable.

> jfs and reiserfs (both 3 and 4) are options, but they are both used so seldom 
> that I would not be comfortable trusting them.

I wouldn't trust reiser4, but reiserfs (version 3) is super stable - nobody
is messing with it much.  The only interesting question is the last bit of
BKL removal.  Otherwise it's rock solid.  We've been using it for the last 8
years, and it's been solid for about the last 5.

> some people consider ext4 well tested enough, and then the debate between it and 
> XFS gets much more interesting.

yes, it does.
 
> Ted Tso will tell you that ext4 is only extensivly tested in the simple cases 
> (single drive, home system type of thing), while XFS has had a lot of attention 
> and testing on very large and sophisticated drive systems, so the larger and 
> more powerful the drive subsystem you have the more likely you are to run into 
> some corner case that the ext4 developers just haven't run into yet, but that 
> the XFS developers have handled.

Our large sophisticated drive systems emulate a single drive well enough that
it works just fine.  We're not running layers of LVM magic though.  All the
funky RAID stuff is handled by hardware.

Bron.