Another 2.4 upgrade horror story

[Sebastian Hagedorn]

Hi,

about three weeks ago we upgraded our Cyrus installation from 2.3.x to 
2.4.16. We were aware of the reindexing issue, so we took precautionary 
measures, but they didn't help a lot. We've got about 7 TB of mail data for 
almost 200,000 mailboxes. We did the upgrade on a Sunday and had told our 
users that mail access wouldn't be possible for the whole day. After the 
actual software upgrade we ran distributed scripts that triggered the index 
upgrades. We started with the largest mailboxes. The idea was that after 
those that took the longest had been upgraded, the rest should be OK 
overnight and early Monday. However, even though our storage infrastructure 
was kept at 99 % I/O saturation, progress was much slower than anticipated.

Ultimately the server was virtually unuseable for the whole Monday and 
parts of Tuesday. The last mailbox was finally upgraded on Thursday, 
although on Wednesday most things were already working normally.

I realize that some of our problems were caused by infrastructure that's 
not up to current standards, but nonetheless I would really urge you to 
never again use an upgrade mechanism like that. Give admins a chance to 
upgrade indexes in the background and over time.

Sebastian
-- 
     .:.Sebastian Hagedorn - RZKR-W (Gebäude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pkcs7-signature
Size: 5313 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20120925/cf108488/attachment.bin