Another 2.4 upgrade horror story
Hagedorn at uni-koeln.de
Tue Sep 25 08:01:56 EDT 2012
about three weeks ago we upgraded our Cyrus installation from 2.3.x to
2.4.16. We were aware of the reindexing issue, so we took precautionary
measures, but they didn't help a lot. We've got about 7 TB of mail data for
almost 200,000 mailboxes. We did the upgrade on a Sunday and had told our
users that mail access wouldn't be possible for the whole day. After the
actual software upgrade we ran distributed scripts that triggered the index
upgrades. We started with the largest mailboxes. The idea was that after
those that took the longest had been upgraded, the rest should be OK
overnight and early Monday. However, even though our storage infrastructure
was kept at 99 % I/O saturation, progress was much slower than anticipated.
Ultimately the server was virtually unuseable for the whole Monday and
parts of Tuesday. The last mailbox was finally upgraded on Thursday,
although on Wednesday most things were already working normally.
I realize that some of our problems were caused by infrastructure that's
not up to current standards, but nonetheless I would really urge you to
never again use an upgrade mechanism like that. Give admins a chance to
upgrade indexes in the background and over time.
.:.Sebastian Hagedorn - RZKR-W (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 5313 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20120925/cf108488/attachment.bin
More information about the Info-cyrus