Cyrus IMAP running in linux: recommended file system

[David Lang]

On Fri, 17 Dec 2010, Bron Gondwana wrote:

> On Thu, Dec 16, 2010 at 06:09:58PM -0800, David Lang wrote:
>> On Thu, 16 Dec 2010, Lucas Zinato Carraro wrote:
>>
>>> I know that this question is controversy and there is no exact answer.
>>>
>>> But  currently what is the best file system in  Linux to handle
>>> thousands of small files?
>>> I have mailboxes with 4Gb. And messages with ~4Kb.
>>> Ext4 ?
>>
>> depending on who you ask you will get
>>
>> ext4
>> btrfs
>> XFS
>>
>> my personal opinion is that ext4 is still new enought that I don't trust it
>> fully (they are still finding too many problems and having to scramble to fix
>> them)
>
> Disagree - it's been quite stable for about a year now, and usable for at
> least two.  The stuff they're finding is skanky little edge cases that
> we haven't run in to at all, so I'm happy.
>
> We're running about 50% on ext4.  About 1% on btrfs (a testing store, no real
> users)

that's very fair. I'm probably biased on the fact that I read the linux-kernel 
list and so see all the 'how in the world is it doing _that_' type of problems 
that come up :-)

>> this makes btrfs much too new.
>
> Agree.
>
>> so I opt for XFS
>
> Performance with zillions of small files isn't XFS's strong suit.

lots of files works fine on XFS, what it is slow with is metadata operations 
(creating or deleting lots of files), delayed expunge hides the latter problem.

David Lang

>> you want to avoid ext2/ext3 as they do very poorly with large numbers of files
>> in one directory.
>
> Definitely agree.  ext3 was unusable.
>
>> jfs and reiserfs (both 3 and 4) are options, but they are both used so seldom
>> that I would not be comfortable trusting them.
>
> I wouldn't trust reiser4, but reiserfs (version 3) is super stable - nobody
> is messing with it much.  The only interesting question is the last bit of
> BKL removal.  Otherwise it's rock solid.  We've been using it for the last 8
> years, and it's been solid for about the last 5.
>
>> some people consider ext4 well tested enough, and then the debate between it and
>> XFS gets much more interesting.
>
> yes, it does.
>
>> Ted Tso will tell you that ext4 is only extensivly tested in the simple cases
>> (single drive, home system type of thing), while XFS has had a lot of attention
>> and testing on very large and sophisticated drive systems, so the larger and
>> more powerful the drive subsystem you have the more likely you are to run into
>> some corner case that the ext4 developers just haven't run into yet, but that
>> the XFS developers have handled.
>
> Our large sophisticated drive systems emulate a single drive well enough that
> it works just fine.  We're not running layers of LVM magic though.  All the
> funky RAID stuff is handled by hardware.
>
> Bron.
>