Latest on Cyrus and GFS?

[Stephen L. Ulmer]

 "jwatkins" == Jared Watkins <jwatkins at snowcrash.homeip.net> writes:

  jwatkins> I've not been able to find the answer to the GFS question
  jwatkins> in the archives.. so here goes.  If one were using a dual
  jwatkins> attach scsi cabinet.. or fibre channel..  would it
  jwatkins> possible to use cyrus on GFS from two or more servers all
  jwatkins> with r/w access?  I'm thinking of a setup that would have
  jwatkins> all user mailboxes visible to all cyrus servers.. but
  jwatkins> clients would be directed to different systems with
  jwatkins> perdition to balance the load.  So if one server needed to
  jwatkins> come down.. you would simply take that system out of the
  jwatkins> pool and direct the users hitting it to the remaining
  jwatkins> cyrus servers.. increasing their load... but leaving the
  jwatkins> clients none the wiser. Jared


We were attempting to use IBM's GPFS to do exactly this.  It isn't
going to work well.

The main problem is that Cyrus makes heavy usage of MMAP, and if you
turn of MMAP in Cyrus, you've still got locking issues within
BerkeleyDB.  I posted a patch to this list a few weeks ago that get
you about 85% of the way to using BerkleyDB's built-in RPC abstraction
for the mailboxes.db, but it's just not going to work as well as
Murder.

There are also uses of MMAP within Cyrus that are looked on as
unconventional.  For example, Cyrus will MMAP a file (for r/w), then
open the same file (from the same process) for writing -- thus
invalidating the map.  In a distributed filesystem, you've locked the
mmap'd region of the file across the cluster -- that invalidation if
*very* expensive.

There will also be regular-old file locking issues.  Since distributed
filesystems are slow with meta-data updates, stat(), fcntl(), and
friends are going to be painful.

A Murder allows segmentation (for most purposes) of the mailboxes.db,
which is the single-most-contended-for resource in Cyrus (as far as I
can tell).  The problem that *isn't* addresses is availability of each
IMAP back-end.  We're hoping to use IBM's HACMP for this, I think that
there is a Linux-ish, scriptable-takeover-on-hearbeat-failure hooya,
though I don't know what it is called.

Can it be done?  Yes.  It won't give you even close to the same
performance of a Murder.  In our environment, it just sucked and we
found out the hard way.


Regards,

--
Stephen L. Ulmer                              ulmer at nersp.nerdc.ufl.edu
Senior Systems Programmer                         http://www.ulmer.org/
Northeast Regional Data Center                      VOX: (352) 392-2061
University of Florida                               FAX: (352) 392-9440