LARGE single-system Cyrus installs?

[Rob Mueller]

> Yesterday I checked my own Cyrus servers to see if I was running out of 
> lowmem, and it sure looked like it.  Lowmem had only a couple MB free, and 
> I had 2GB of free memory that was not being used for cache.
>
> I checked again today and everything seems to be fine - 150MB of lowmem 
> free and almost no free memory (3GB cached)!  Grrr.

I'm guessing it can depend on recent operations. From my understanding, the 
issue is that inode structures can only be in lowmem, but actual cache pages 
can be anywhere, but you can only have cached file pages if you also have 
the corresponding inode structure. So if you have lots of small files (eg 
like cyrus), then you can run out of lowmem for inode structures before you 
run out of total memory for cached pages.

However if your access pattern changes, and you access fewer but larger 
files, then lowmem might be big enough to hold enough inodes to use up the 
reset of memory for cache pages. For instance if you've just run cyr_expire, 
it would have read a lot cyrus.cache files, which are a lot bigger than your 
average email, so in that case you might end up with all memory used by 
cache but with lowmem still free.

One thing we've found very useful is munin, which has allowed us to track 
patterns and long term changes over time. The graphs in the blog post are 
directly from munin. If you can get it setup on your machine, I recommend it 
as a nice visual way of getting to know your machines over time.

http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/

In the above case, if my hypothesis were true, you'd be able to see it 
easily on a munin graph by comparing the memory graph with the open_inodes 
graph and your known cyr_expire run times to see if there is a relationship.

> Anyways, I was looking into building a 64-bit kernel.  I'm running Debian 
> Sarge (I know, old) on a Dell 2850 with Intel Xeon (Nocona) CPUs and 4GB 
> RAM.  My kernel version is 2.6.14.5, built from kernel.org sources.  It 
> has "High Memory Support (64GB)" selected.
>
> When I run menuconfig, I'm not seeing any obvious place to switch from 
> 32-bit to 64-bit.  Could you elaborate a bit about how you switched to a 
> 64-bit kernel?  Also, are you running a full 64-bit distro, or just a 
> 64-bit kernel?

Bron did all the OS changes. We're running a 64bit kernel on a 32bit system, 
it was a lot easier than upgrading the entire system. I believe debian makes 
this pretty easy to do (we use etch though, not sarge).

I've added him to the cc list so he can reply with the details.

Rob