Painful performance on mupdate push (how long does yours take?)

[Andrew Morgan]

On Tue, 26 May 2009, Michael Bacon wrote:

> We're in the last stages of getting off of our old single-server mail
> system into a murder environment, and we're currently at the point
> where the single old backend gets starts pushing its mb database to
> the mupdate server, so that we can move the main DNS pointer to the
> front-ends and start moving mailboxes over to other servers.
>
> Last Friday night, I tried that push for the first time, thinking it
> would take 10-20 minutes, tops (for an 850k line mailbox database, or
> about 75 MB).  3.5 hours later, it finally completed, and a subsequent
> ctl_mboxlist -m hung indefinitely, with neither side doing anything
> (as per truss).
>
> Since this was originally a push to a non-global Solaris zone
> connecting to a ZFS file system over the "local loopback" interface
> that Sun has for its zones, I tried to see if the problem went away by
> connecting to a vanilla Solaris install on UFS.  Nope, still syncing
> on the order of 30-40k per minute.  I tried shutting off the GSSAPI
> security layer (max SASL layer set to 0), same performance issue.
> Since all of this was on Sun CoolThreads hardware, I spent the last
> part of this afternoon trying to run it on an old school sparc box,
> but didn't get it running before I left for the day.
>
> Surely this isn't the kind of performance I should expect, right?  For
> large sites, how long does an initial mboxlist push to the MUPDATE
> server usually take, and for how many lines?  And is anyone else
> running a murder on Sun CoolThreads (Tx000, T5x20) server hardware?

Which database format are you using for mailboxes.db?  I found that 
skiplist was very slow for writes compared to Berkeley DB.  However this 
push is only rarely done so it was still worth avoiding the hassle of 
Berkeley DB.

 	Andy