Cyrus upgrades and then -- DB corruption

[Patrick Boutilier]

On 12/05/2009 05:15 PM, Jack Neely wrote:
> Folks,
>
> We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL
> 3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus
> version 2.3.14.
>
> I've had 2 cases now where the upgrade appeared to go smoothly.  Cyrus
> comes up, logs look fine, lmtp mail is flowing in, users are happy, and
> the logs are perfect.  For a few minutes until the alians corrupt the
> database.  Suddenly we see stuff like:
>
> Dec  5 15:02:55 xxxxxx lmtp[4724]: DBERROR: error fetching
> <9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>:
> DB_PAGE_NOTFOUND: Requested page not found
>
> Dec  5 15:02:55 xxxxxxxxx lmtp[4724]: DBERROR: mystore: error storing
> <9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>:
> DB_PAGE_NOTFOUND: Requested page not found
>
> Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: page 0: illegal page type or format
> Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: PANIC: Invalid argument
> Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR: critical database situation
>
> It goes downhill from there.  LMTP doesn't deliever, the IMAP port stops
> responding.  DBERRORs scroll.
>
> I could not dump the mailboxes.db with
>
> su cyrus -c "/local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db">  /tmp/mailboxes.corrupt
>
> Just more errors.  I could not move mailboxes.db out of the way and load
> a known good dump back in its place....more dberrors.
>
> Finally, I moved the db/ directory and created an empty db/ as well as
> moved the deliever.db out of the way (some lmtp errors were citing that
> file.)  I turned her back on and she's now up and running like nothing
> happened.
>
> Can someone help me understand what happened here and how to avoid it?

Remove /var/imap/db/* (keep the directory) and get rid of 
/var/imap/db.backup* directories before starting Cyrus for the first 
time after upgrading to 2.3.x .

At least that is what I did when upgrading from Fedora Core 2 with 2.2.x 
Cyrus to CentOS 5.x with Cyrus 2.3.x to avoid those errors. Got them 
when testing the upgrade on a separate machine.

>
> Thanks,
> Jack Neely