*ouch* Cyrus just died on me: DBERROR: opening /var/imap/mailboxes.db: Not enough space

[Nikos Voutsinas]

Paul Boven wrote:
> Hi everyone,
> 
> Last evening Cyrus died on me. Fortunately all messages were queued and
> nothing was lost, but I can't make sense of the error-messages.
> And I'd like to know, in order to ensure this doesn't happen again.
> 
> System is a SunFire V120, Solaris 9, Cyrus 2.1.10 (same one I messaged a 
> few days ago regarding the performance problems with ctl_cyrusdb)

The "xxxx.db: Not enough space" has nothing to do with the core cyrus 
code, or the mailboxes.db.  It's rather a problem of DB 4.x.x 
/cyrusdb_db3, and appears when the  DB backend operates under a heavy 
load. (that's why this problem usually appears when the db is the 
choosen backend for mailboxes.db). Unfortunately It is not clear to me 
(yet) if this is only a DB's bug or not, but I am working on it.

If this makes you feel better take a look to the following logs. (Note 
that the appearing "ldapcache.db" is a non-cyrus code, that is build on 
DB 4.1.25-p1 and cyrusdb_db3, and is also pushing the DB backend )

imapd[21656]: [ID 818864 local6.info] get_values: cache miss
imapd[21654]: [ID 729713 local6.error] DBERROR: opening 
/opt/Cyrus/var/ldapcache.db: cyrusdb error
imapd[21654]: [ID 388555 local6.error] cyrus_ldap_cache error: can't 
initialize cache db
imapd[21654]: [ID 381270 local6.info] get_ldaphost: returns 
195.134.65.215 195.134.65.224
imapd[21656]: [ID 381270 local6.info] get_ldaphost: returns 
195.134.65.215 195.134.65.224
imapd[21662]: [ID 729713 local6.error] DBERROR: opening 
/opt/Cyrus/var/ldapcache.db: Not enough space
imapd[21662]: [ID 729713 local6.error] DBERROR: opening 
/opt/Cyrus/var/ldapcache.db: cyrusdb error
imapd[21662]: [ID 388555 local6.error] cyrus_ldap_cache error: can't 
initialize cache db
imapd[21662]: [ID 381270 local6.info] get_ldaphost: returns 
195.134.65.224 195.134.65.215
imapd[21661]: [ID 729713 local6.error] DBERROR: opening 
/opt/Cyrus/var/ldapcache.db: Not enough space
imapd[21661]: [ID 729713 local6.error] DBERROR: opening 
/opt/Cyrus/var/ldapcache.db: cyrusdb error
imapd[21661]: [ID 388555 local6.error] cyrus_ldap_cache error: can't 
initialize cache db

> 
>  From /var/log/imapd.log:
> master[9764]: [ID 392559 local6.debug] about to exec 
> /usr/local/cyrus/bin/lmtpd
> lmtpunix[9764]: [ID 518349 local6.debug] executed
> lmtpd[9764]: [ID 729713 local6.error] DBERROR: opening 
> /var/imap/mailboxes.db: Not enough space

This message usually means that an underlying Berkeley DB
shared memory region is too small. (  __db.00[12345]). Try this one 
:db_stat -e -h path/to/var/db

> lmtpd[9764]: [ID 729713 local6.error] DBERROR: opening 
> /var/imap/mailboxes.db: cyrusdb error
> lmtpd[9764]: [ID 157947 local6.error] FATAL: can't read mailboxes file
> lmtpd[9764]: [ID 866726 local6.warning] DBERROR db4: Database handles 
> open during environment close

What this means is you have DB handles that you have opened
in a database environment and never closed, and that's not
allowed when you call the enclosing DB_ENV->close method.

> lmtpd[9764]: [ID 688962 local6.error] DBERROR: error exiting 
> application: Invalid argument
> lmtpd[9764]: [ID 688962 local6.error] DBERROR: error exiting 
> application: cyrusdb error
> master[623]: [ID 310780 local6.debug] process 9764 exited, status 75
> 
> I can't explain the error-message 'not enough space'. I have the 
> cricket-graphs here, and the machine never lacked diskspace or memory, 
> in fact it has never paged since it was turned on.
> 
> Any help greatly appreciated.
> 
> Regards, Paul Boven.

Nikos Voutsinas