Another 2.4 upgrade horror story

[Janne Peltonen]

Hi!

On Tue, Sep 25, 2012 at 05:55:27PM +0200, Wolfgang Breyha wrote:
> Time was not the limiting factor for us. Availability and safety of our
> mailboxes was. Nobody can guarantee you that the migration works out
> flawlessly. If moving one mailbox fails I have troubles with exactly one
> mailbox. If reconstruction of one mailbox fails while migrating hard from 2.3
> to 2.4 your system is down even longer. And we had a couple of problematic
> mailboxes as partly documented on this mailinglist;-)

Um. Isn't it still only that one mailbox that couldn't be reconstructed that's
unaccessible, with the system as a whole up and running and everything else
accessible and working? Or am I missing something? I mean, each mailbox gets
recontsructed separately. And by having tested the migration, we had reason to
believe most if not all of the mailboxes would get reconstructed OK, so we
didn't expect that any significant number of users would see any trouble except
maybe slowness for a short while.

As it happened, we didn't have any trouble with the reconstruction of any
mailboxes, the process went flawlessly - it was just slow. Also, it did cause
observable slowness only because at first, I hadn't reconstructed the mailboxes
systematically but had trusted the system to do it on demand. After the
business day and daily peak usage was over, I set up such a number of
reconstruction processes that the system didn't choke on them, and everything
(~60k mailboxes) was reconstructed before next morning.

> In detail I
> *) check for active sessions on the mailbox
> *) lock the mailbox by denying access via userdeny.db
> *) move the mailbox
> *) unlock it

Yeah, that process is identical to the one we use to move mailboxes between
backends to balance their disk usage. (The mailbox-moving part of the process,
that is; the algorithm to decide how many mailboxes to move from where and
whence is another matter.)


--Janne
-- 
Janne Peltonen <janne.peltonen at helsinki.fi> PGP Key ID: 0x9CFAC88B
Please consider membership of the Hospitality Club (http://www.hospitalityclub.org)