painful mupdate syncs between front-ends and database server

[Andrew Morgan]

On Mon, 19 Oct 2009, Michael Bacon wrote:

> Hello, list,
>
> Today we're enjoying our first full work day of independence from the old
> monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU
> boards since then, but that's it), and on our new shiny cluster of T5220's
> that are mostly happily operating as a murder.
>
> I say mostly because while most of the times the thing handles our 80,000
> users and 14,000+ simultaneous connections like a champ, some of the time,
> we get some extreme pain, mostly due to syncs between the MUPDATE master
> and the front-end servers.
>
> When we spec'ed out our servers, we didn't put much I/O capacity into the
> front-end servers -- just a pair of mirrored 10k disks doing the OS, the
> logging, the mailboxes.db, and all the webmail action going on in another
> solaris zone on the same hardware.  We thought this was sufficient given
> the fact that no real permanent data lives on these servers, but it turns
> out that while most of thie time it's fine, if the mupdate processes ever
> decide they need to re-sync with the master, we've got 6 minutes of trouble
> ahead while it downloads and stores the 800k entries in the mailboxes.db.

What is causing a (re)sync of the frontends?  Normally this should only 
happen when you start Cyrus on a frontend, right?

> During these sync periods, we see two negative impacts.  The first is
> lockup on the mailboxes.db on the front-end servers, which slows down both
> accepting new IMAP/POP connections and the reception of incoming messages.
> (The front-ends also accept LMTP connections from a separate pair of
> queueing hosts, then proxy those to the back-ends.)  The second is that,
> because the front-ends go into a

A part of this paragraph was chopped off.  What else did you have to say?

I ran some tests back in 2006:

> However, I just performed an interesting test comparing skiplist versus
> berkeley.
>
> skiplist - approx 20-25mins
> berkeley - 3mins
>
> Those are the times it took to push the entire mailbox list from our test
> server to the mupdate master (146382 mailboxes).

This was a test run populating the mupdate master mailboxes.db from a 
single backend server.  However, I think it illustrates the differences 
between skiplist and berkeley database formats.  In our case, we still 
went with skiplist, but it might be something to consider.

 	Andy