Cyrus crashed again badly

[Florin Andrei]

Running cyrus-imapd-2.2.3 on Linux Fedora Core 1, with Postfix-2.0.16
It's a small system, i'm the only user, mail gets delivered via SMTP to
Postfix locally, which delivers it to Cyrus. I'm also using Sieve.
Evolution is the principal client, with some webmailers used
occasionally.
The load is not much, i'm a very busy email user (bout 1k messages /
day), but still i'm only one person, and i'm the sole user of that
system, so the system is quite idle.

This is the second time the Cyrus IMAP server crashes spectacularly.

First time, months ago, i was able to somehow regenerate mailboxes.db -
don't ask me how, i only remember i found a script on the Internet that
creates a text version of it which can be imported. Even so, since that
crash it wouldn't let me create new folders. But that was ok,
temporarily.

Now it crashed again. I've found a huge email queue in the local Postfix
server this morning - all messages waiting to be delivered to me
locally. According to the logs, it happened Saturday early in the
morning (that's when the messages started to queue up), now it's Monday
at noon and everything is broken.
I tried to run reconstruct, ctl_cyrusdb, ctl_cyrusdb_all, i moved away
deliver.db temporarily... nothing helped.

I cannot read my email, i have no idea how to fix it. I'm in a pretty
bad situation. I'm writing this message from my other (personal)
account.
The cyrus documentation is not very helpful either.

Please understand that i'm making efforts to stay rational. I'll provide
you guys with all information about my system that you will require, i'm
willing to run scripts and whatnot. I hope we could work together to fix
it.
But having the IMAP server crash twice in a few months with no apparent
reason, and not having documentation available for what to do in such
situations - it's not encouraging. In any case, i'm still willing to put
up with it and do my best to fix it, provided that i get some kind of
guidance.

Logs are below. Please let me know what else you need from me. Thank you
in advance.

##########################################################
May 10 12:07:24 stantz master[19933]: process started
May 10 12:07:24 stantz master[19934]: about to exec
/usr/lib/cyrus-imapd/ctl_cyrusdb
May 10 12:07:26 stantz ctl_cyrusdb[19934]: DBERROR db4: Improper file
close at 5/9465323
May 10 12:07:26 stantz ctl_cyrusdb[19934]: DBERROR db4: Recovery
function for LSN 5 9465323 failed on backward pass
May 10 12:07:26 stantz ctl_cyrusdb[19934]: DBERROR db4: PANIC: Invalid
argument
May 10 12:07:26 stantz ctl_cyrusdb[19934]: DBERROR: critical database
situation
May 10 12:07:26 stantz master[19933]: process 19934 exited, status 75
May 10 12:07:26 stantz master[19937]: about to exec
/usr/lib/cyrus-imapd/idled
May 10 12:07:26 stantz idled[19937]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:26 stantz idled[19937]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:26 stantz idled[19937]: DBERROR: init() on berkeley
May 10 12:07:26 stantz master[19933]: ready for work
May 10 12:07:26 stantz master[19942]: about to exec
/usr/lib/cyrus-imapd/ctl_cyrusdb
May 10 12:07:26 stantz ctl_cyrusdb[19942]: DBERROR db4: fatal region
error detected; run recovery
May 10 12:07:26 stantz ctl_cyrusdb[19942]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:26 stantz ctl_cyrusdb[19942]: DBERROR: init() on berkeley
May 10 12:07:26 stantz master[19943]: about to exec
/usr/lib/cyrus-imapd/imapd
May 10 12:07:26 stantz master[19944]: about to exec
/usr/lib/cyrus-imapd/imapd
May 10 12:07:26 stantz master[19945]: about to exec
/usr/lib/cyrus-imapd/lmtpd
May 10 12:07:26 stantz ctl_cyrusdb[19942]: checkpointing cyrus databases
May 10 12:07:26 stantz imap[19943]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:26 stantz imaps[19944]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:26 stantz lmtpunix[19945]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:26 stantz ctl_cyrusdb[19942]: archiving database file:
/var/lib/imap/annotations.db
May 10 12:07:26 stantz imap[19943]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:26 stantz imaps[19944]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:27 stantz lmtpunix[19945]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR db4: txn_checkpoint
interface requires an environment configured for the transaction
subsystem
May 10 12:07:27 stantz imap[19943]: DBERROR: init() on berkeley
May 10 12:07:27 stantz imaps[19944]: DBERROR: init() on berkeley
May 10 12:07:27 stantz lmtpunix[19945]: DBERROR: init() on berkeley
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR: couldn't checkpoint:
Invalid argument
May 10 12:07:27 stantz imap[19943]: executed
May 10 12:07:27 stantz imaps[19944]: executed
May 10 12:07:27 stantz lmtpunix[19945]: executed
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR: sync
/var/lib/imap/db: cyrusdb error
May 10 12:07:27 stantz lmtpunix[19945]: DBERROR db4: environment not yet
opened
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR db4:
DB_ENV->log_archive interface requires an environment configured for the
logging subsystem
May 10 12:07:27 stantz lmtpunix[19945]: DBERROR: opening
/var/lib/imap/deliver.db: Invalid argument
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR: error listing log
files: Invalid argument
May 10 12:07:27 stantz lmtpunix[19945]: DBERROR: opening
/var/lib/imap/deliver.db: cyrusdb error
May 10 12:07:27 stantz ctl_cyrusdb[19942]: DBERROR: archive
/var/lib/imap/db: cyrusdb error
May 10 12:07:27 stantz lmtpunix[19945]: FATAL: lmtpd: unable to init
duplicate delivery database
May 10 12:07:28 stantz ctl_cyrusdb[19942]: archiving database file:
/var/lib/imap/mailboxes.db
May 10 12:07:28 stantz master[19933]: process 19945 exited, status 75
May 10 12:07:28 stantz ctl_cyrusdb[19942]: DBERROR db4: txn_checkpoint
interface requires an environment configured for the transaction
subsystem
May 10 12:07:28 stantz master[19933]: service lmtpunix pid 19945 in
READY state: terminated abnormally
May 10 12:07:28 stantz ctl_cyrusdb[19942]: DBERROR: couldn't checkpoint:
Invalid argument
May 10 12:07:28 stantz master[19946]: about to exec
/usr/lib/cyrus-imapd/imapd
May 10 12:07:28 stantz master[19947]: about to exec
/usr/lib/cyrus-imapd/lmtpd
May 10 12:07:28 stantz ctl_cyrusdb[19942]: DBERROR: sync
/var/lib/imap/db: cyrusdb error
May 10 12:07:28 stantz imap[19946]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:28 stantz lmtpunix[19947]: DBERROR db4: fatal region error
detected; run recovery
May 10 12:07:28 stantz ctl_cyrusdb[19942]: DBERROR db4:
DB_ENV->log_archive interface requires an environment configured for the
logging subsystem
May 10 12:07:28 stantz imap[19946]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:28 stantz lmtpunix[19947]: DBERROR: dbenv->open
'/var/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
recovery
May 10 12:07:28 stantz master[19933]: exiting on SIGTERM/SIGINT
########################################################

-- 
Florin Andrei

http://florin.myip.org/

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html