DB4 hosed on production server: Please help!!

[Nels Lindquist]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Our production Cyrus server is down; please CC replies directly to me 
at "nels at maei.ca" -- I can still get that mail, while my subscribed 
address mailstore is on the affected server.

I tried a routine restart of our IMAP server this morning, but it 
would appear that db4 is toast, and ctl_cyrusdb isn't helping.

I looked back in the logs and noticed these messages which first 
appeared yesterday morning:

> Apr 21 03:34:13 aerodrome ctl_cyrusdb[23220]: DBERROR db4:
> DB_LOGC->get: invalid log record header 

And here's what happened when I tried to restart the server:

> Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> DB_ENV->log_flush: LSN past current end-of-log 
> Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> /usr/lib/imap/deliver.db: unable to flush page: 0 
> Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> txn_checkpoint: failed to flush the buffer cache Invalid argument 
> Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR: couldn't
> checkpoint: Invalid argument 
> Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR: sync
> /usr/lib/imap/db: cyrusdb error 
> Apr 22 10:07:36 aerodrome master[28548]: setrlimit: Unable to set file
> descriptors limit to 2147483647: Operation not permitted 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> DB_ENV->log_flush: LSN past current end-of-log 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> /usr/lib/imap/deliver.db: unable to flush page: 0 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> txn_checkpoint: failed to flush the buffer cache Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4: PANIC:
> Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR: critical
> database situation 
> Apr 22 10:07:36 aerodrome master[28548]: process 28550 exited, status
> 75 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4: fatal
> region error detected; run recovery 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: dbenv->open
> '/usr/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
> recovery 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: init() on
> berkeley 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> txn_checkpoint interface requires an environment configured for the
> transaction subsystem 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: couldn't
> checkpoint: Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: sync
> /usr/lib/imap/db: cyrusdb error 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> DB_ENV->log_archive interface requires an environment configured for
> the logging subsystem 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: error listing
> log files: Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: archive
> /usr/lib/imap/db: cyrusdb error 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> txn_checkpoint interface requires an environment configured for the
> transaction subsystem 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: couldn't
> checkpoint: Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: sync
> /usr/lib/imap/db: cyrusdb error 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> DB_ENV->log_archive interface requires an environment configured for
> the logging subsystem 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: error listing
> log files: Invalid argument 
> Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: archive
> /usr/lib/imap/db: cyrusdb error 

Running "ctl_cyrusdb -r" manually as the cyrus user does the same 
thing.

How can I fix this and get my server back up?  How can I prevent this 
from happening again?

Please help!

- ----
Nels Lindquist <*>
Information Systems Manager
Morningstar Air Express Inc.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)

iD8DBQFAh/q+bxRqvNchgLQRApzxAJ9hvbGOhd95b6kDy4vF94B+zl3JVACgofJo
gSjCHoz0a5kbpfiCB3ocJ5U=
=YdhC
-----END PGP SIGNATURE-----
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html