Upgrade failure from 2.2.1 to 2.2.8

[Scott Adkins]

Okay, so I have just attempted for a second time to upgrade our Cyrus server
on the production box from version 2.2.1 to version 2.2.8.  The upgrade was
a failure and I had to back out and go through a not-quite-painless recovery
process on the user accounts affected by the upgrade.

So, the first time I did this we ran into two obvious bugs on the Tru64 5.x
platform.  The first bug was that after a bit, the processes leaked all
their file descriptors and eventually starting crapping out.  In a previous
e-mail, I described this as a problem with getnameinfo() on Tru64 and we
will likely have to work this problem through HP's tech support.

The second bug was tied to some user's sieve scripts failing.  In another
previous e-mail, I described that bug and supplied a fix.

What was most problematic about that upgrade was that quite a number of
users ended up with corrupted cyrus.header and cyrus.index files and had
to be reconstructed.  The users reported various problems, such as opening
their mailboxes and finding no messages in them, even though they knew
there were... or finding messages, but when opening a message, it would
be blank when they knew it wasn't.  Reconstructing the accounts fixed the
problems.

So, after fixing the above bugs, we attempted another upgrade tonight.  I
started seeing a lot of log messages from LMTP to the following effect:

    verify_user(user.cyrus) failed: Mailbox has an invalid format

The server ran for nearly 20 minutes.  in that time, there were 1717 of
these errors affecting 136 different users.  I believe that this is related
in some way to the mailbox corruption during the previous upgrade.

The archives suggest reconstructing accounts that show this, and maybe
we would be okay by continuing to run on the new version and reconstructing
all existing accounts, but we have over 93,000 accounts and over 376,000
mailboxes... so, reconstructing everything is not the way to go.

I read the upgrade instructions and the only things mentioned when going
from 2.2.1 to 2.2.8 is the bytecode format change (i.e. recompiling all the
user sieve scripts) and the backend database stuff is now specified in the
imapd.conf file (which I did and double-checked to make sure that what is
supplied matches what I had compiled in the 2.2.1 version of the server).
However, the cyrus.header and cyrus.index files are not related to any of
the backend database stuff, so I don't understand what is going on here.

Does anyone have any ideas on how to troubleshoot this?  I guess the first
thing I need to do is to duplicate it in our test environment... All the
testing I have done never once triggered this problem.

Anyways, I hope somebody can help me on this.

Thanks,
Scott
-- 
 +-----------------------------------------------------------------------+
      Scott W. Adkins                http://www.cns.ohiou.edu/~sadkins/
   UNIX Systems Engineer                  mailto:adkinss at ohio.edu
        ICQ 7626282                 Work (740)593-9478 Fax (740)593-1944
 +-----------------------------------------------------------------------+
     PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 231 bytes
Desc: not available
Url : https://lists.andrew.cmu.edu/mailman/private/info-cyrus/attachments/20040830/602a6a05/attachment.bin