DB4 hosed on production server: Please help!!

[Andrew Morgan]

Can you just delete/move the deliver.db and restart Cyrus?  That is not a
critical database, considering that you are currently down.

	Andy

On Thu, 22 Apr 2004, Nels Lindquist wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Our production Cyrus server is down; please CC replies directly to me
> at "nels at maei.ca" -- I can still get that mail, while my subscribed
> address mailstore is on the affected server.
>
> I tried a routine restart of our IMAP server this morning, but it
> would appear that db4 is toast, and ctl_cyrusdb isn't helping.
>
> I looked back in the logs and noticed these messages which first
> appeared yesterday morning:
>
> > Apr 21 03:34:13 aerodrome ctl_cyrusdb[23220]: DBERROR db4:
> > DB_LOGC->get: invalid log record header
>
> And here's what happened when I tried to restart the server:
>
> > Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> > DB_ENV->log_flush: LSN past current end-of-log
> > Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> > /usr/lib/imap/deliver.db: unable to flush page: 0
> > Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR db4:
> > txn_checkpoint: failed to flush the buffer cache Invalid argument
> > Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR: couldn't
> > checkpoint: Invalid argument
> > Apr 22 10:06:55 aerodrome ctl_cyrusdb[28465]: DBERROR: sync
> > /usr/lib/imap/db: cyrusdb error
> > Apr 22 10:07:36 aerodrome master[28548]: setrlimit: Unable to set file
> > descriptors limit to 2147483647: Operation not permitted
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> > DB_ENV->log_flush: LSN past current end-of-log
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> > /usr/lib/imap/deliver.db: unable to flush page: 0
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4:
> > txn_checkpoint: failed to flush the buffer cache Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR db4: PANIC:
> > Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28550]: DBERROR: critical
> > database situation
> > Apr 22 10:07:36 aerodrome master[28548]: process 28550 exited, status
> > 75
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4: fatal
> > region error detected; run recovery
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: dbenv->open
> > '/usr/lib/imap/db' failed: DB_RUNRECOVERY: Fatal error, run database
> > recovery
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: init() on
> > berkeley
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> > txn_checkpoint interface requires an environment configured for the
> > transaction subsystem
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: couldn't
> > checkpoint: Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: sync
> > /usr/lib/imap/db: cyrusdb error
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> > DB_ENV->log_archive interface requires an environment configured for
> > the logging subsystem
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: error listing
> > log files: Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: archive
> > /usr/lib/imap/db: cyrusdb error
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> > txn_checkpoint interface requires an environment configured for the
> > transaction subsystem
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: couldn't
> > checkpoint: Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: sync
> > /usr/lib/imap/db: cyrusdb error
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR db4:
> > DB_ENV->log_archive interface requires an environment configured for
> > the logging subsystem
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: error listing
> > log files: Invalid argument
> > Apr 22 10:07:36 aerodrome ctl_cyrusdb[28556]: DBERROR: archive
> > /usr/lib/imap/db: cyrusdb error
>
> Running "ctl_cyrusdb -r" manually as the cyrus user does the same
> thing.
>
> How can I fix this and get my server back up?  How can I prevent this
> from happening again?
>
> Please help!
>
> - ----
> Nels Lindquist <*>
> Information Systems Manager
> Morningstar Air Express Inc.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (MingW32)
>
> iD8DBQFAh/q+bxRqvNchgLQRApzxAJ9hvbGOhd95b6kDy4vF94B+zl3JVACgofJo
> gSjCHoz0a5kbpfiCB3ocJ5U=
> =YdhC
> -----END PGP SIGNATURE-----
> ---
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html