LARGE single-system Cyrus installs?

[Rob Mueller]

> A data point regarding reiserfs/ext3:
>
> We are in the process of moving from reiserfs to ext3 (with dir_index).
>
> ext3 seems to do substantially better than reiserfs for us, especially for
> read heavy loads (squatter runs at least twice as fast as it used do).

Are you comparing an "old" reiserfs partition with a "new" ext3 one where 
you've just copied the email over to? If so, that's not a fair comparison.

What we found was that when we first copied the data from reiserfs -> ext3, 
everything seemed nice, but that's only because when you copy mailboxes 
over, you're effectively defragmenting all the files and directories in your 
filesystem. We've actually been thinking of setting up a regular process to 
just randomly move users around, because the act of moving them effectively 
defragments them regardless of the filesystem you're using!

Give it a month or two of active use though (delivering new emails, deleting 
old ones, etc), and everything starts getting fragmented again. Then ext3 
really started going to crap on us. Machines that had been absolutely fine 
under reiserfs, the load just blew out to unuseable under ext3.

> data=ordered in both cases. data=journal didn't seem to make any
> difference with ext3. data=journal with reiserfs caused amusing kernel
> memory leaks, which it looks like Fastmail also hit recently. An dedicated
> journal device would probably make a big difference with data=journal.

Talking with Chris Mason about this, data=journal is faster in certain 
scenarios with lots of small files + fsyncs from different processes, 
exactly the type of workload cyrus generates!

As it turns out, the memory leaks weren't critical, because the the pages do 
seem to be reclaimed when needed, though it was annoying not knowing exactly 
how much memory was really free/used. The biggest problem was that with 
cyrus you have millions of small files, and with a 32bit linux kernel the 
inode cache has to be in low memory, severely limiting how many files the OS 
will cache.

See this blog post for the gory details, and why a 64-bit kernel was a nice 
win for us.

http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/

Rob