Cyrus upgrades and then -- DB corruption

[Jack Neely]

On Sat, Dec 05, 2009 at 06:35:43PM -0400, Patrick Boutilier wrote:
> On 12/05/2009 05:15 PM, Jack Neely wrote:
> > Folks,
> >
> > We are continuing our quest to upgrade our Cyrus IMAP servers from RHEL
> > 3 i386 with Cyrus version 2.2.12 to RHEL 5 x86_64 machines running cyrus
> > version 2.3.14.
> >
> > I've had 2 cases now where the upgrade appeared to go smoothly.  Cyrus
> > comes up, logs look fine, lmtp mail is flowing in, users are happy, and
> > the logs are perfect.  For a few minutes until the alians corrupt the
> > database.  Suddenly we see stuff like:
> >
> > Dec  5 15:02:55 xxxxxx lmtp[4724]: DBERROR: error fetching
> > <9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>:
> > DB_PAGE_NOTFOUND: Requested page not found
> >
> > Dec  5 15:02:55 xxxxxxxxx lmtp[4724]: DBERROR: mystore: error storing
> > <9311515.102504011260020246443.JavaMail.em-build at na-mm2-relay.amazon.com>:
> > DB_PAGE_NOTFOUND: Requested page not found
> >
> > Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: page 0: illegal page type or format
> > Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR db4: PANIC: Invalid argument
> > Dec  5 15:03:55 xxxxxx lmtp[4724]: DBERROR: critical database situation
> >
> > It goes downhill from there.  LMTP doesn't deliever, the IMAP port stops
> > responding.  DBERRORs scroll.
> >
> > I could not dump the mailboxes.db with
> >
> > su cyrus -c "/local/cyrus/sbin/ctl_mboxlist -d -f /imap/conf/mailboxes.db">  /tmp/mailboxes.corrupt
> >
> > Just more errors.  I could not move mailboxes.db out of the way and load
> > a known good dump back in its place....more dberrors.
> >
> > Finally, I moved the db/ directory and created an empty db/ as well as
> > moved the deliever.db out of the way (some lmtp errors were citing that
> > file.)  I turned her back on and she's now up and running like nothing
> > happened.
> >
> > Can someone help me understand what happened here and how to avoid it?
> 
> Remove /var/imap/db/* (keep the directory) and get rid of 
> /var/imap/db.backup* directories before starting Cyrus for the first 
> time after upgrading to 2.3.x .
> 
> At least that is what I did when upgrading from Fedora Core 2 with 2.2.x 
> Cyrus to CentOS 5.x with Cyrus 2.3.x to avoid those errors. Got them 
> when testing the upgrade on a separate machine.
> 

That's just it...the old databases are never present.  I dump
mailboxes.db on the old server just like above, scp that to the new
server on new storage, and use that to recreate a new mailboxes.db.
None of the other databases are copied over in any way.

Cyrus does seem to regenerate all the databases propperly...and most of
the time it stays working without corruption.  Most of the time.  ;-)

Thanks for your help!
Jack Neely

-- 
Jack Neely <jjneely at ncsu.edu>
Linux Czar, OIT Campus Linux Services
Office of Information Technology, NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89