constant cyrus failures with DBERROR db4

[Simon Matter]

>
> We are having constant problems with cyrus locking up and filling
> up the system log with gigabytes of errors like this:
>
> Sep  8 13:07:09 mail master[23769]: process 21100 exited, status 75
> Sep  8 13:07:09 mail master[23769]: service imaps pid 21100 in READY
> state:
> terminated abnormally
> Sep  8 13:07:09 mail lmtpunix[21102]: DBERROR db4: PANIC: fatal region
> error detected; run recovery
> Sep  8 13:07:09 mail lmtpunix[21102]: DBERROR: critical database situation
> Sep  8 13:07:09 mail master[21103]: about to exec
> /usr/lib64/cyrus-imapd/imapd
> Sep  8 13:07:09 mail master[21104]: about to exec
> /usr/lib64/cyrus-imapd/imapd
> Sep  8 13:07:09 mail pop3[21101]: DBERROR db4: PANIC: fatal region error
> detected; run recovery
> Sep  8 13:07:09 mail pop3[21101]: DBERROR: critical database situation
> Sep  8 13:07:09 mail master[21105]: about to exec
> /usr/lib64/cyrus-imapd/pop3d
> Sep  8 13:07:09 mail master[21106]: about to exec
> /usr/lib64/cyrus-imapd/pop3d
> Sep  8 13:07:09 mail imap[21103]: DBERROR db4: PANIC: fatal region error
> detected; run recovery
>
> till of course the / partition is full and then all hell breaks loose.
>
> We can get things going again by doing the following which I found
> via a search on the web:
>
> service cyrus-imapd stop
> /bin/rm /var/lib/imap/db/log.*
> su - cyrus -c "/usr/lib64/cyrus-imapd/ctl_cyrusdb -r"
> service cyrus-imapd start
>
> It is unclear to me what if anything is lost by doing the above.
> Anyone know?
>
> But is there a fix for this?  We are running 2.2.12-3.RHEL4.1 on a
> CentOS4.1 x86_64 box

You may want to switch berkeley dbs to skiplist to try how it goes.
By default only duplicate_db and tlscache_db are berkeley so you may try
this in /etc/imapd.conf:
duplicate_db: skiplist
tlscache_db: skiplist

One special note:
The cyrus-imapd package you are using (the package from RHEL4 is based on
my Invoca package) tries to convert the db files automatically on startup.
However, the version you have has a bug which prevents it from converting
all db files correctly. So you may want to convert the existing db files
with cvt_cyrusdb by hand to avoid problems - or update your package to my
last one from http://www.invoca.ch/pub/packages/cyrus-imapd/ which has
several other bugs fixed.

And then, please if you switch and enjoy skiplist, let us know how it
works for you and how it performs compared to before, and let us know
something about your user count, concurrent sessions and such. I'm still
waiting for reports of people who decided to eliminate the berkeley DB
problems by switching to skiplist.

Regards,
Simon

>
> Is there a way to at least just make cyrus die instead of filling
> up the log with the DBERROR messages.
>
> This has happened twice today, and two other times in past 40 days.
>
> --
> ---------------------------------------------------------------
> Paul Raines                email: raines at nmr.mgh.harvard.edu
> MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging
> 149 (2301) 13th Street     Charlestown, MA 02129	    USA
>
>
> ----
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
>