Fwd: Docs for Cyrus/GFS?

[Jens Hoffrichter]

There I told him to write to the list, and I just clicked the wrong
reply button myself ;)

Sorry for that.


---------- Forwarded message ----------
From: Jens Hoffrichter <jens.hoffrichter at gmail.com>
Date: 2008/9/8
Subject: Re: Docs for Cyrus/GFS?
To: "Chris St. Pierre" <stpierre at nebrwesleyan.edu>


Hi Chris,

2008/9/8 Chris St. Pierre <stpierre at nebrwesleyan.edu>:

> I'm looking for any good documentation on using Cyrus IMAP with shared
> storage (in our case, GFS).  Thus far, all I've been able to turn up
> is a few snippets on mailing lists and in the wiki, but nothing really
> comprehensive.  Any pointers?  Thanks!
There is no real documentation up to now, I think, but I'm in the
process of implementing a quite big mail system on top of a GFS, so if
you have any specific questions, just ask here on the list or me
directly, though on the list would probably be better, to share the
answers.

Just some hints for things I stumbled upon up to now:

I suppose you meta directory will be shared, so you need to separate
out some directories between the single nodes, where each node can put
its own files. Directories coming to my mind are "proc" and "backup".
There are probably some more, but I have no access to the machine
right now to check it. For the separation there I used the named
symlinks from GFS, which worked quite fine.

The other thing is: Make ABSOLUTELY sure that you turn off anything
related to bdb during compile (you probably need to compile your own
cyrus anyway, we are using the incova sources here), as berkeley db
just doesn't work on top of GFS. Or, at least not in a cyrus context.
And it isn't sufficient to just use skiplist in the config file, you
definitely need to switch off compiling bdb support.

The reason for this is that even if you don't have any bdbs
configured, cyrus will still initialize some sort of bdb environment
in your meta directory, and if on another sort the same thing is
happening during the same time, it will try to sleep on a mutex,
waiting for the release of the mutex. Yet, the callback to wake up
never comes when using GFS, so your process will hang indefinitely.
That one took us at least 3 weeks to figure out ;)

Besides from those little things, cyrus works on top of the GFS just
fine, you probably need to tweak some tuning parameters, but I don't
remember them right now, if you need them, I can look them up for you.

The performance is nothing to brag about, though, cyrus and GFS just
don't play that well together. But I got the performance I needed
(around 15 logins per second with three nodes using a loadbalancer),
which is sufficient for our needs. The added reliability is nice,
though, as you can turn off one node, and the whole thing still works.

I don't know if you have any experience using RedHat Cluster, but be
absolutely sure to understand about fencing and quorum of the cluster
before you start any configuration. Cluster software is by no means
really easy to use, IMO, and everything you do needs some extra
thinking if you are just used working with single nodes normally :)

Just try your luck, and if you have any questions, just ask.

Jens