Cyrus IMAP and MySQL mailboxes (Building load-balancing cluster)

[Igor Zhbanov]

2006/11/16, adam at morrison-ind.com <adam at morrison-ind.com>:
>
> > The short question is: can Cyrus IMAP take mail from MySQL tables?
>
> I'm not aware of any such backend;  and I think it would be a crazy
> method to store mail and performance would suck.  You may *THINK*
> MySQL is fast, it isn't - certainly not compared to a filesystem
> [doubly so when dealling with BLOBs,  which is essentially what a mail
> message is]
>
> > And where can I read about how to setup it?
>
> There is a package called DbMail.  Maybe you want to look at that.
>
> > The long story is now. I need to setup load-balancing mail system. I
> > culdn't find open-source mail systems that have genuine cluster
> > support,

> What is "genuine cluster support"?
I mean some support of clustering by software itself

> > The main problem is to build a shared storage that can survive server
> > crashes, where mail will be stored.
>
> Use a SAN.
First of all, such SAN must be very reliable itself. Second, it must
support some kind of global locking mechanism, so several nodes can
use lock to protect file from simultaneous writing. Third, Cyrus IMAP
must lock mailboxes, so several instances on different server can work
with one mailbox without conflicts. Whether Cyrus IMAP use locks or
assumes that he is the only one who access mail box, I don't know. Can
it safely access one mailbox from different servers, I don't know too.

> > I have found that MySQL-cluster is
> > reliable and fast. So, mail will be stored in MySQL tables. And I know
> > that Postfix can store mail in MySQL. So, I need POP3 server that can
> > take letters from MySQL tables.
>
> See DbMail.

---

2006/11/17, Bron Gondwana <brong at fastmail.fm>:
> On Thu, Nov 16, 2006 at 07:24:32PM +0300, Igor Zhbanov wrote:
> > The main problem is to build a shared storage that can survive server
> > crashes, where mail will be stored. I have found that MySQL-cluster is
> > reliable and fast.
>
> My
>
> god.
>
> You are aware that MySQL-cluster only supports in-memory databases in
> all but the most recent development snapshots?
>
> If your mail system is so small you can afford to put all your users
> email in memory, then good for you.  Otherwise, mysql replication won't
> buy you much more than Cyrus replication with a few good monitoring
> scripts (and yes, we have failed real cyrus replications off failed
> machines now - it's never fun, but going through the logs we lost a
> grand total of two messages, and they had both been sieved into the
> Junk Mail folder anyway.)

Yes, I know. But the latest version of MySQL can use on disk
(non-indexed) tables fields. I don't think that I need search by
letter content. If I will need it (of course, via web-interface), I
think I can make that feature.

> Seriously, see the other response, DbMail might be what you want -
> personally I'd put blobs in the filesystem (actually, my SHA1 based
> VFS system, but that's a different story) and metadata in mysql... if
> I was writing my perfect IMAP solution, which I'm not, yet.  Cyrus
> does the job just fine, and you work around the wrinkles.  It's better
> than anything else out there for a biggish system right now.

I will look at DbMail too. Generally, I don't need exactly MySQL. I
just want to build load-balanced mail system. Probably, based on
Postfix + Cyrus IMAP + Squirrel Mail. I have found some links that may
be useful but didn't look at them yet.

http://cyrusimap.web.cmu.edu/ag.html
http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/replication.html
http://cyrusimap.web.cmu.edu/imapd/install-replication.html

First of all, Cyrus IMAP has feature called "Cyrus IMAP Aggregator":
"The Cyrus IMAP Aggregator transparently distributes IMAP and POP
mailboxes across multiple servers. Unlike other systems for load
balancing IMAP mailboxes, the aggregator allows users to access
mailboxes on any of the IMAP servers in the system." Probably, it's
what I want.

Also, there is Cyrus IMAP replication.But some people says that there
can be lost letters during failover. And, of course, "use at your own
risk".

For now I see two ways to build load-balanced mail system:
1) Some kind of shared storage. It may be NFS-like global filesystem
or MySQL database.
2) Mailbox replication. It can be done by Cyrus IMAP replication or by
some other software.

Perhaps, there are other ways. I will look...

Thanks.