choosing a file system

[Martin Wendel]

Bron Gondwana wrote:
> On Thu, Jan 08, 2009 at 10:13:25PM -0800, Robert Banz wrote:

>> (notice, didn't mention AIX. I've got my standards ;)
> 
> Hey - I have a friend who _likes_ AIX.  There are odd people in the
> world.

We at Uppsala university have been running cyrus on AIX for a little more than 10
years. Back then, there was no acceptable alternative to the AIX LVM. AIX still is,
as I see it, very competent when it comes to handling disk I/O. About three years
ago I thought time was ready for running cyrus on a shared file system. We had a
large installation of IBM SANFS on another system that at the time was performing
well. We purchased SANFS and four RS/6000 servers for cyrus and needed to take it
in production rather abruptly after a filesystem crash on the old cyrus server.

However time was not quite ready for running cyrus on SANFS. After about two years
of SANFS problems (although at no time did SANFS get corrupt, it was very stable
but just could not handle the load. Sometimes we needed to restart the filesystem
several times a week) we decided to follow the examples published on this list by the
FastMail guys. I did not dare to go with GPFS when SANFS was discontinued.

We've since splitted our 20 million cyrus files onto eight IBM blade servers, running
AIX virtualization server handling SAN connections for the 6 virtual RedHat servers
running on each of them. We run one cyrus instance on each RedHat server, 3 primary
servers and 3 replicas on each blade. We thus have 24 primary servers, and 24 replicas,
with about 1 million cyrus files each.

We did some tests on which file system to choose but there were not that much difference
so we decided on ext3.

We also have 4 additional blades running Debian, 2 for LVS and 2 for Nginx, and about
10TB of SAN disk area dedicated to cyrus. The system has been running very nicely for six
months now.

So I guess this is a success story inspired by FastMail. But I still would not choose
anything other than AIX for our TSM servers.