The Cyrus db (/var/imap/db) is suddenly hundreds of MB!

[Paul Boven]

Hi everyone,

Due to a problem with a webserver on the same machine, the /var
partition on our mailserver went from over 1GB available to full within
a few minutes.

The Cyrus database lives in /var/imap, and has become corrupt as a
result of this. Initially, we tried to recover from this situation by
stopping all services and moving the offending Apache logfiles away.

But as soon as we made room on /var, Cyrus would use it all up in
/var/imap/db which is now 863MB in size. What's inside:

-rw-------   1 cyrus    mail        8192 Oct  6 17:14 __db.001
-rw-------   1 cyrus    mail      663552 Oct  6 17:14 __db.002
-rw-------   1 cyrus    mail       98304 Oct  6 17:13 __db.003
-rw-------   1 cyrus    mail     21061632 Oct  6 17:13 __db.004
-rw-------   1 cyrus    mail       32768 Oct  6 17:13 __db.005
-rw-------   1 cyrus    mail     10485738 Oct  6 15:58 log.0000000357
-rw-------   1 cyrus    mail     10485728 Oct  6 15:58 log.0000000358
-rw-------   1 cyrus    mail     10485728 Oct  6 15:58 log.0000000359
(85 of these logfiles in total)
-rw-------   1 cyrus    mail           4 Sep 30 18:28 skipstamp

Trying to run 'ctl_cyrusdb -r' takes a few minutes but aborts with a
coredump. When I try to start Cyrus, it tries to recover the database
(but coredumps), then starts copying all these 10MB logfiles into
db.backup and we run out of diskspace again.

Any hints on how to get this system back into working order would be
very much appreciated!

Regards, Paul Boven.

Some background info:
Solaris 9 USparc
Cyrus-Imap 2.2.8
Cyrus-Sasl2 2.1.19

/etc/cyrus.conf:
START { recover         cmd="ctl_cyrusdb -r" }

SERVICES {
imap    cmd="imapd" listen="imap" prefork=0
sieve   cmd="timsieved" listen="sieve" prefork=0
lmtpunix      cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0

}

EVENTS {
checkpoint    cmd="ctl_cyrusdb -c" period=30
delprune      cmd="cyr_expire -E 3" at=0400
tlsprune      cmd="tls_prune" at=0400 }

/etc/imapd.conf
configdirectory: /var/imap
partition-default: /var/spool/imap
admins: cyrus
sasl_pwcheck_method: saslauthd
allowplaintext: yes
sieveusehomedir: false
sievedir: /var/spool/sieve
altnamespace: yes
virtdomains: no
hashimapspool: 1
fulldirhash: 1
createonpost: 1
autocreatequota: -1
sasl_auxprop_plugin: plain digestmd5
sasl_sql_select: dummy
allowusermoves: yes

Backtrace from the coredump:
core 'core' of 6551:    /usr/local/cyrus/bin/ctl_cyrusdb -r
 ff159520 __dbreg_open_file (fedec, d4899, d6978, e0468, 0, d5c58) + f8
 ff159084 __dbreg_register_recover (d5c58, 0, ffbfe8e0, 6, 0, ff1cfc6c)
+ 1cc
 ff140594 __db_dispatch (d5c58, e0468, b6, ffbfe9ac, ffbfe8e0, 6) + 128
 ff15f218 __env_openfiles (d5c58, d7410, e0468, ffff8755, a00000, 1) + f8
 ff15e428 __db_apprec (d5c58, ffbfe988, 20, ff1cfc6c, ff1cfc6c, 0) + 5f8
 ff15be44 __dbenv_open (d5cf8, 4000, d5cfc, 4000, 80020, d5c58) + 540
 00037114 init     (0, d4000, ffbfebbd, ffbfebb0, 7000, 36f34) + 1e0
 00030500 cyrusdb_init (e, 64, b0, 141, 29c98, c8fe8) + 6c
 0001f0e0 cyrus_init (0, 3a6a0, 3bc00, 0, 0, 1) + 3cc
 00015c2c main     (2, ffbffd24, ffbffd30, 3a660, 0, d4000) + 108
 00015830 _start   (0, 0, 0, 0, 0, 0) + 5c