High availability email server...

Mon Jul 31 17:00:04 EDT 2006

Okay, okay, I just can't *NOT* say something here :)

First, I disagree with all the statements below.  Cyrus CAN run in an
Active/Active mode, you CAN have multiple servers reading and writing to
the same files, and clustering IS a good way to achieve HA/DR/BC in a
Cyrus environment.  Why do I say that?  I say that because we have been
doing that for many many years.

The key is to have a GOOD clustering file technology that does proper
file locking while still providing good performance.  For years, we have
been running our Cyrus IMAP system on a Tru64 Alpha TruCluster system.
Our cluster has 4 members in it, 2 of which serves up Cyrus IMAP/IMSP,
and the other 2 which accepts and delivers mail via Sendmail.  All of
the servers have access to the full set of files across the cluster and
everything "just works".

I would like to address a few specific comments listed below:

  > GFS, Lustre, and other cluster filesystems do file-level locking; in
  > order to properly read and write to the BDB backend, you'd need DB-level
  > locking, which is not possible from a filesystem.

I wrote a lengthy response to this, but when I got the conclusion, it all
came down to a really simple point.  How is having multiple servers any
different than a single server?  You still have tons of different processes
all trying to acquire read/write locks to the same files.  There is no one
process in Cyrus that opens the database and shares it with all the other
processes running under Cyrus.  How is this different from an IMAP process
running on one server and a different IMAP process running on a different
server?  There isn't.  The end result is that file locking is the most
important feature Cyrus has to rely upon... what if you are using the flat
file format for your mailboxes.db file?  At that point, that is the ONLY
thing you can rely upon...

  > IMAP is also a stateful connection; depending on how you set up your
  > cluster, some clients might not handle it gracefully (e.g., Pine).

True, true... stateful it is... but at the same time, what kind of problems
do you see?  When an IMAP connection is opened, the client auths and starts
doing stuff with it.  When it closes the connection, it is done with it.
That is it.  If another connection is opened, presumably, it is because of
a user initiating a new action, and the client will simply do the same thing
again... auth and work.  Most clients keep at least one connection open to
the server at all times.  Even if the client has more than one connection,
and one connection is on one server and another connection is on another
server, there still shouldn't be any problems.... The data is the same on
the other end.  Incidentally, we have Pine users in our environment that do
not have problems with our multi-server clustering Cyrus environment.  In
fact, we have not seen any client have problems with it.

Webmail based clients are a different animal.  It isn't because of the fact
that we are running multi-servers in the environment, it is becasue of the
non-stateful nature of the client.  Users don't have problems with antyhing
from a data consistency standpoint, it is simply a problem with performance.
It is the same issue faced in a single server environment.  Using some kind
of middleware piece to cache IMAP connections is usually how this problem is
solved.

  > As already said in this thread: Cyrus cannot share its spool.
  > No 2 cyrus instances can use the same spool, databases and lockfiles.

Just simply isn't true.  However, I must say, there is a reason why NFS
shouldn't be used... it doesn't do proper file locking (though, I am going
to watch for the responses on the NFSv4 thread that somebody asked about).
Without proper file locking, even a single Cyrus server on the backend is
jeapordized by multiple IMAP processes wanting to write to a single DB at
the same time.

  > Clustered filesystems don't make any sense for Cyrus, since the
  > application itself doesn't allow simultaneous read/write.

I completely disagree... Clustering filesystems (if they implement proper
file locking techniques) actually SIMPLIFIES your setup significantly.  You
don't have to have a complex Murder/Perdition environment with replication,
failover, etc.  You simply run 2 or more servers running on the clustering
filesystem and run things as you would normally expect.  Surprisingly, it
runs quite well.

And finally:

  > Anyway, it has nothing to do with Cyrus, but if anyone does have
  > another application that wants lots of small files on a clustered FS:
  >
  > <http://web.caspur.it/Files/2005/01/10/1105354214692.pdf>
  > <http://polyserve.com/pdf/Caspur_CS.pdf>

Kinda surprising, but it DOES have something to do with Cyrus.  Caspur
did their case study on cluster filesystems with their e-mail environment.
It used Cyrus IMAP and some kind of SMTP (I think it was Postfix or
something like that), since that was the e-mail environment they had.
Polyserve came out as the big winner.  It is very good reading.  It is a
good case where a clustering filesystem was specifically chosen to handle
their Cyrus e-mail environment.

Incidentally, for those who care, we are planning on migrating our own
Cyrus environment out of Tru64 into RedHat running on Polyserve by the end
of the year (hopefully).

Scott

--On Monday, July 31, 2006 8:05 AM -0500 "Chris St. Pierre" <stpierre at NebrWesleyan.edu> wrote:

> One of the major problems you'd run into is /var/lib/imap, the config
> directory.  It contains, among other things, a Berkeley DB of
> information about the mail store.  GFS, Lustre, and other cluster
> filesystems do file-level locking; in order to properly read and write
> to the BDB backend, you'd need DB-level locking, which is not possible
> from a filesystem.  If you tried putting /var/lib/imap on shared
> storage, you'd have data corruption and loss in no time.
>
> IMAP is also a stateful connection; depending on how you set up your
> cluster, some clients might not handle it gracefully (e.g., Pine).

--On Saturday, July 29, 2006 12:40 PM +0200 Daniel Eckl <deckl at nero.com> wrote:

> As already said in this thread: Cyrus cannot share its spool.
> No 2 cyrus instances can use the same spool, databases and lockfiles.
>
> For load balancing you can use a murder setup and for HA you can use
> replication.

--On Friday, July 28, 2006 3:52 PM -0500 Rich Graves <rgraves at carleton.edu> wrote:

> Clustered filesystems don't make any sense for Cyrus, since the application
> itself doesn't allow simultaneous read/write. Just use a normal journaling
> filesystem and fail over by mounting the FS on the backup server. Consider
> replication such as DRDB or proprietary SAN replication if you feel you
> must physically mirror the storage.
>
> Anyway, it has nothing to do with Cyrus, but if anyone does have another
> application that wants lots of small files on a clustered FS:
>
> <http://web.caspur.it/Files/2005/01/10/1105354214692.pdf>
> <http://polyserve.com/pdf/Caspur_CS.pdf>

-- 
 +-----------------------------------------------------------------------+
      Scott W. Adkins                http://www.cns.ohiou.edu/~sadkins/
   UNIX Systems Engineer                  mailto:adkinss at ohio.edu
        ICQ 7626282                 Work (740)593-9478 Fax (740)593-1944
 +-----------------------------------------------------------------------+
     PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 229 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20060731/ce62a2e4/attachment.bin