ReiserFS and general cyrus filesystem usage information - was Re: best filesystem for imap server

Thu Dec 2 15:48:02 EST 2004

On Thu, 02 Dec 2004, Rob Mueller wrote:
> We use reiserfs for our large cyrus installation. We changed from ext3 
[...]

That was very interesting and useful data, thanks for posting it!

> Ordered = Data is written before meta-data journal is committed. This 
> avoids filesystem and data corruption. This is now the default in >= 2.6.8.1
> Data = All data and meta-data is written to the journal

Ordered would be best for a Cyrus spoll, and I guess Data would be best on
MTAs (when they have a small enough queue lifetime for most messages, and
the journal is large enough).

> turned out that's not the major IO bottleneck. We've found that the 
> mailboxes.db, .seen and quota databases generate the most IO. Putting these 

Indeed. Although why mailboxes.db (when using the BDB backend, anyway) has
so much IO I have no idea.  Once read, BDB should be doing IPC to fetch it
from in-memory cache, not trashing the disk.  Unless writes to mailboxes.db
are very common.

> One other useful feature of reiserfs is the "tails" feature. This is on by 
> default, and it means that multiple small files can be stored in 1 disk 
> block. On a space limited nvram drive, this is very useful for the legacy 
> quota system which uses 1 file small file per quota root (eg usually per 

No doubts about that one (since we're talking about a nvram drive here).  I
wonder if it is such a great idea on a slow device (disk), though.  Do you
have this data?

> wouldn't be appropriate. We could have used bdb, but generally have had 
> lots of problems with bdb so don't entirely trust it...

I don't know of anyone sane that trusts any BDB on the 4.x series.

> I should add potential problem as well. There appears to be an issue on 
> heavily loaded linux servers with the way the the cyrus skiplist db works. 
[...]
> problem existed in ext3 as well 
> (http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/0966.html). It seems 
> this is a very rare problem though since no-one else has reported it. There 
> are patches available to fix both in case anyone else has come across it.

You mean the patches on the threads, or patches available somewhere else
(where?)

> All up, we've been very happy with reiserfs and i'd recommend people use 
> it, especially in >= 2.6.8.1 kernels where data=ordered is now the default 
> option.

How stable and stress-tested is data=ordered? and what about the full
journalling (which might be a good thing on MTAs)?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html