Cyrus upgrades and then -- DB corruption

[Jack Neely]

Folks,

We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL
3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus
version 2.3.14.

I've had 2 cases now where the upgrade appeared to go smoothly.  Cyrus
comes up, logs look fine, lmtp mail is flowing in, users are happy, and
the logs are perfect.  For a few minutes until the alians corrupt the
database.  Suddenly we see stuff like:

Dec  5 15:02:55 xxxxxx lmtp[4724]: DBERROR: error fetching
<9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>:
DB_PAGE_NOTFOUND: Requested page not found

Dec  5 15:02:55 xxxxxxxxx lmtp[4724]: DBERROR: mystore: error storing 
<9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>: 
DB_PAGE_NOTFOUND: Requested page not found

Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: page 0: illegal page type or format
Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: PANIC: Invalid argument
Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR: critical database situation

It goes downhill from there.  LMTP doesn't deliever, the IMAP port stops
responding.  DBERRORs scroll.

I could not dump the mailboxes.db with

su cyrus -c "/local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db" > /tmp/mailboxes.corrupt

Just more errors.  I could not move mailboxes.db out of the way and load
a known good dump back in its place....more dberrors.

Finally, I moved the db/ directory and created an empty db/ as well as
moved the deliever.db out of the way (some lmtp errors were citing that
file.)  I turned her back on and she's now up and running like nothing
happened.

Can someone help me understand what happened here and how to avoid it?

Thanks,
Jack Neely
-- 
Jack Neely <jjneely at ncsu.edu>
Linux Czar, OIT Campus Linux Services
Office of Information Technology, NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89