my cyrus went crazy - "PANIC: DB_RUNRECOVERY: Fatal error" and others

Sat Mar 19 18:12:52 EST 2005

Hello,

my environment:

Gentoo
net-mail/cyrus-imapd 2.2.12
sys-libs/db 4.1.25_p1-r4 1.85-r2

today in the morning I have noticed that postfix cannot deliver mail to 
mailboxes. A fast look at /var/log/everything/* showed that Cyrus went 
crazy with messages like these:

Mar 19 12:29:33 [lmtpunix] DBERROR db4: environment not yet opened
Mar 19 12:29:33 [lmtpunix] DBERROR: opening /var/imap/deliver.db: 
Invalid argument
Mar 19 12:29:33 [lmtpunix] DBERROR: opening /var/imap/deliver.db: 
cyrusdb error
Mar 19 12:29:33 [lmtpunix] FATAL: lmtpd: unable to init duplicate 
delivery database
Mar 19 12:29:33 [master] service lmtpunix pid 30445 in READY state: 
terminated abnormally

unfortunately logs have already rotated so many times, that could not 
find information about the beginning of the disaster.
Anyway - /etc/init.d/cyrus restart fortunately helped and until now 
everything was fine.

But then... cyrus went down again with logs like this:

Mar 19 22:27:28 [postfix/pipe] 86247530B5: to=<grzes at pyza.from.cx>, 
orig_to=<root at pyza.from.cx>, relay=cyrus, delay=1, status=sent 
(pyza.from.cx)
Mar 19 22:27:29 [postfix/pipe] 86247530B5: to=<marcin at pyza.from.cx>, 
orig_to=<root at pyza.from.cx>, relay=cyrus, delay=2, status=sent 
(pyza.from.cx)
Mar 19 22:27:29 [postfix/qmgr] 86247530B5: removed

(till now it was ok)

Mar 19 22:30:01 [/usr/sbin/cron] (root) CMD (test -x /usr/sbin/run-crons 
&& /usr/sbin/run-crons )
Mar 19 22:32:15 [ctl_cyrusdb] checkpointing cyrus databases
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: DB_LOGC->get: log record 
checksum mismatch
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: DB_LOGC->get: catastrophic 
recovery may be required
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: PANIC: DB_RUNRECOVERY: Fatal 
error, run database recovery
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR: critical database situation
Mar 19 22:33:20 [imap] DBERROR db4: fatal region error detected; run 
recovery
Mar 19 22:33:20 [imap] DBERROR: dbenv->open '/var/imap/db' failed: 
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:33:20 [imap] DBERROR: init() on berkeley
Mar 19 22:33:20 [imap] sql_select option missing
Mar 19 22:33:20 [imap] auxpropfunc error no mechanism available_
Mar 19 22:33:20 [imap] login: localhost [127.0.0.1] marcin plaintext 
User logged in
Mar 19 22:38:21 [imap] DBERROR db4: fatal region error detected; run 
recovery

and then:

Mar 19 22:51:05 [imaps] idle for too long, closing connection
                 - Last output repeated twice -
Mar 19 22:52:05 [imaps] DBERROR db4: fatal region error detected; run 
recovery
Mar 19 22:52:05 [imaps] DBERROR: error closing: DB_RUNRECOVERY: Fatal 
error, run database recovery
Mar 19 22:52:05 [imaps] DBERROR: error closing tlsdb: cyrusdb error
Mar 19 22:52:05 [imaps] DBERROR db4: fatal region error detected; run 
recovery
Mar 19 22:52:05 [imaps] DBERROR: error exiting application: 
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:52:13 [imaps] DBERROR db4: fatal region error detected; run 
recovery
Mar 19 22:52:13 [imaps] DBERROR: error closing: DB_RUNRECOVERY: Fatal 
error, run database recovery
Mar 19 22:52:13 [imaps] DBERROR: error closing tlsdb: cyrusdb error
Mar 19 22:52:13 [imaps] DBERROR db4: fatal region error detected; run 
recovery
Mar 19 22:52:13 [imaps] DBERROR: error exiting application: 
DB_RUNRECOVERY: Fatal error, run database recovery
(I use imaps for remote mail clients)

I tried to restart cyrus, but got only:

Mar 19 22:59:57 [master] exiting on SIGTERM/SIGINT
Mar 19 22:59:58 [master] setrlimit: Unable to set file descriptors limit 
to -1: Operation not permitted
Mar 19 22:59:58 [master] retrying with 1024 (current max)
Mar 19 22:59:58 [master] process started
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_flush: LSN past 
current end-of-log
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: /var/imap/deliver.db: unable 
to flush page: 0
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: txn_checkpoint: failed to 
flush the buffer cache Invalid argument
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: PANIC: Invalid argument
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR: critical database situation
Mar 19 22:59:58 [master] process 3885 exited, status 75_
Mar 19 22:59:59 [master] ready for work
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: fatal region error detected; 
run recovery
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: dbenv->open '/var/imap/db' 
failed: DB_RUNRECOVERY: Fatal error, run database  recovery
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: init() on berkeley
Mar 19 22:59:59 [ctl_cyrusdb] checkpointing cyrus databases
Mar 19 22:59:59 [tls_prune] DBERROR db4: fatal region error detected; 
run recovery
Mar 19 22:59:59 [tls_prune] DBERROR: dbenv->open '/var/imap/db' failed: 
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:59:59 [tls_prune] DBERROR: init() on berkeley
Mar 19 22:59:59 [tls_prune] DBERROR db4: environment not yet opened
Mar 19 22:59:59 [tls_prune] DBERROR: opening /var/imap/tls_sessions.db: 
Invalid argument
Mar 19 22:59:59 [tls_prune] DBERROR: opening /var/imap/tls_sessions.db: 
cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: txn_checkpoint interface 
requires an environment configured for the tran
saction subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: couldn't checkpoint: Invalid argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: sync /var/imap/db: cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_archive interface 
requires an environment configured for the logging subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: error listing log files: Invalid 
argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: archive /var/imap/db: cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: txn_checkpoint interface 
requires an environment configured for the tran
saction subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: couldn't checkpoint: Invalid argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: sync /var/imap/db: cyrusdb error
[...]

I tried ctl_cyrusdb -r, with no effect :

Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_flush: LSN past 
current end-of-log
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: /var/imap/deliver.db: unable 
to flush page: 0
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: txn_checkpoint: failed to 
flush the buffer cache Invalid argument
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: PANIC: Invalid argument
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR: critical database situation

Please help!
_________________
Rgds, Grzes
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html