Another 2.4 upgrade horror story

[Simon Beale]

> Hi!
>
> On Tue, Sep 25, 2012 at 03:25:45PM +0200, Wolfgang Breyha wrote:
>> Sebastian Hagedorn wrote, on 25.09.2012 14:01:
>> > I realize that some of our problems were caused by infrastructure
>> that's
>> > not up to current standards, but nonetheless I would really urge you
>> to
>> > never again use an upgrade mechanism like that. Give admins a chance
>> to
>> > upgrade indexes in the background and over time.
>>
>> There is such an upgrade path using a murder environment and moving
>> mailboxes
>> between backends. We used that for our 150k user infrastructure and had
>> no IO
>> headaches at all. It was a good moment to update distribution,
>> filesystems,
>> hardware, .... as well.
>
> Could you elaborate on that? I considered that option, but seeing as
> moving
> even a couple dozen users from a backend to another using RENAME takes
> hours
> and one backend contains thousands of users, I decided to just live with
> the ~1
> day of unbearable slowness. Or do you know of a fast way?

I did the migration by moving mailboxes between backends of 1TB each,
having scripted it up to only move employees when it was 00:00 - 06:00 in
their local timezone, and left the script running on each v2.3 backend for
a few days. Took a few weeks in all to migrate and upgrade all our
backends in turn, but no one experienced any downtime.

The only gotcha I experienced was I forgot that cyrus was configured to
hardlink mail, which of course was no longer the case after each mailbox
was migrated, so my disk usage exploded. (But easily fixed/restored once
identified).

It comes down to having spare backend(s) to move people on to, and time to
do it patiently.

Simon