Does Cyrus benefit greatly from increased FS buffer cache?

[Andrew Morgan]

On Tue, 14 Apr 2009, John Widera wrote:

> Hi,
>
> Last week we posted to the list for info about running Cyrus on 64-bit
> RHEL.  But we also needed to ask a more direct question about caching,
> though, and didn't.  So we are now...
>
> That q. is, does Cyrus actually benefit from large amounts of memory in an
> environment with just a few thousand users ( <1000 simultaneous sessions)
> and, say, up to 750GB of mail spool?
>
> Our plan is to throw 12-16GB at it, with the purpose of vastly increasing
> the FS buffer cache (and decreasing I/O).  Or, will that just be a waste
> of RAM?
>
> Some indications are that, yes, it does improve performance notably:
>
> http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/
>
> Anyone have any specific thoughts?  Is there any other benefit we might
> see from large memory allocation in 64-bit architecture?

Funny you mention this, because I was just looking at the load average 
charts we maintain for our cyrus servers.  Over the christmas break, I 
installed a 64-bit kernel (the Debian amd64 kernel) on our backends, which 
each have 4GB of memory.  This is a 32-bit architecture install running a 
64-bit kernel.  The load average on each box is HALF what it was before 
the upgrade.

I know that it is hard for a 32-bit kernel to cache a large number of 
small files, which is frequently the case with cyrus.  It has something to 
do with the size of the lookup table growing too large for "lowmem" to 
contain.  So even if you have lots of memory in a server, it may not be 
able to use it all for file caching if you are trying to cache a lot of 
small files.  The 64-bit kernel does not have a split lowmem/highmem 
architecture, so it can cache a lot more files.

Here is the output of "free" on one of our cyrus backends right now:

              total       used       free     shared    buffers     cached
Mem:       4051576    4024116      27460          0     414200    2703212
-/+ buffers/cache:     906704    3144872
Swap:      2000052        660    1999392

You can see the performance difference on our Cacti graphs at:

   http://admin.onid.oregonstate.edu/cacti/graph.php?rra_id=all&local_graph_id=31

I would definately recommend installing a 64-bit kernel.

 	Andy