Upgrade failure from 2.2.1 to 2.2.8

[Ken Murchison]

Scott Adkins wrote:

> Okay, so I have just attempted for a second time to upgrade our Cyrus 
> server
> on the production box from version 2.2.1 to version 2.2.8.  The upgrade was
> a failure and I had to back out and go through a not-quite-painless 
> recovery
> process on the user accounts affected by the upgrade.
> 
> So, the first time I did this we ran into two obvious bugs on the Tru64 5.x
> platform.  The first bug was that after a bit, the processes leaked all
> their file descriptors and eventually starting crapping out.  In a previous
> e-mail, I described this as a problem with getnameinfo() on Tru64 and we
> will likely have to work this problem through HP's tech support.
> 
> The second bug was tied to some user's sieve scripts failing.  In another
> previous e-mail, I described that bug and supplied a fix.
> 
> What was most problematic about that upgrade was that quite a number of
> users ended up with corrupted cyrus.header and cyrus.index files and had
> to be reconstructed.  The users reported various problems, such as opening
> their mailboxes and finding no messages in them, even though they knew
> there were... or finding messages, but when opening a message, it would
> be blank when they knew it wasn't.  Reconstructing the accounts fixed the
> problems.
> 
> So, after fixing the above bugs, we attempted another upgrade tonight.  I
> started seeing a lot of log messages from LMTP to the following effect:
> 
>    verify_user(user.cyrus) failed: Mailbox has an invalid format
> 
> The server ran for nearly 20 minutes.  in that time, there were 1717 of
> these errors affecting 136 different users.  I believe that this is related
> in some way to the mailbox corruption during the previous upgrade.
> 
> The archives suggest reconstructing accounts that show this, and maybe
> we would be okay by continuing to run on the new version and reconstructing
> all existing accounts, but we have over 93,000 accounts and over 376,000
> mailboxes... so, reconstructing everything is not the way to go.
> 
> I read the upgrade instructions and the only things mentioned when going
> from 2.2.1 to 2.2.8 is the bytecode format change (i.e. recompiling all the
> user sieve scripts) and the backend database stuff is now specified in the
> imapd.conf file (which I did and double-checked to make sure that what is
> supplied matches what I had compiled in the 2.2.1 version of the server).
> However, the cyrus.header and cyrus.index files are not related to any of
> the backend database stuff, so I don't understand what is going on here.
> 
> Does anyone have any ideas on how to troubleshoot this?  I guess the first
> thing I need to do is to duplicate it in our test environment... All the
> testing I have done never once triggered this problem.

It sounds like the on-the-fly upgrade of cyrus.index and cyrus.cache is 
broken when going to 2.2.1 to 2.2.8.  I've looked at the code and 
nothing seems obvious (yet), so if you can duplicate it and possibly 
post before and after versions of these files, it might help track down 
the problem.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html