cyrus-murder problems with database corruption in the frontend/master

[João Assad]

Come on guys, someone must have at least an idea I can try.
Anything will help, maybe Im missing something obvious.

---- a repost of the problem below, a bit more elaborated.

We use cyrus-imapd-murder as the solution for our website messaging/e-mail
service meaning all our users have an e-mail account they can use to 
exchange
messages within the site. Its only accessible through the site and doesnt
directly exchange e-mail with the outside world. Pretty much like a webmail
interface.

We currently have 1.200.000+ mailboxes split in 3 backend servers using 1
frontend / master server (both services running on the same server) for a
grand total of 3.400.000+ mailboxes and subfolders.

the servers are used for imap access only. no pop3, no sieve scripts,
no nntp, no shared folders , nothing else besides imap.

Now, the problem in question.

after some days running smoothly I start getting errors on the  
master/frontend
server. It has happened a few times now..

the errors:

   cyrus/mupdate[1433]: DBERROR: skiplist recovery
   /var/lib/imap/mailboxes.db: 141BAD94 should be ADD or DELETE
   cyrus/mupdate[1433]: DBERROR: error updating database user.1215407: 
cyrusdb error

then after some time I start getting these:

   cyrus/lmtp[1276]: authentication to remote mupdate server failed: EOF 
from server
   cyrus/lmtp[1276]: couldn't connect to 10.1.5.101: no authentication 
to server

after a little more time

   cyrus/imap[30969]: kick_mupdate: can't connect to target: Connection 
refused
   cyrus/lmtp[2240]: mupdate-client: connection to server closed: end of 
file reached
   cyrus/lmtp[2240]: couldn't connect to 10.1.5.101: no connection to server

then this is what I get when trying to restart cyrus frontend/master

   cyrus/ctl_cyrusdb[30607]: recovering cyrus databases
   cyrus/ctl_cyrusdb[30607]: DBERROR: skiplist recovery 
/var/lib/imap/mailboxes.db: 141BAD94 should be ADD or DELETE
   cyrus/ctl_cyrusdb[30607]: DBERROR: opening 
/var/lib/imap/mailboxes.db: cyrusdb error


what I need to do next is completely delete the dbs from the master/frontend
and reimport all the mailboxes, but that takes hours. I came up with a 
faster
solution wich is to configure a dummy backend on the same server the cyrus
frontend/master instance runs which reduces the import time by hours. Any
sujestions on this matter would also be appreciated.

Thank you very much,

João Assad

------------------
Below is my frontend/master configuration and cyrus packages info is on 
the bottom.

##### cyrus.conf

   START {
    # do not delete this entry!
    recover       cmd="ctl_cyrusdb -r"
   }

   # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
   SERVICES {
    # add or remove based on preferences
    mupdate       cmd="mupdate -m" listen=3905 prefork=1
    lmtp          cmd="lmtpproxyd" listen="lmtp" prefork=0
    imap          cmd="proxyd" listen="imap" prefork=0
   }

   EVENTS {
    # this is required
    checkpoint    cmd="ctl_cyrusdb -c" period=240
   }

###### imapd.conf

   configdirectory: /var/lib/imap
   partition-default: /tmp
   admins: cyrus
   sievedir: /var/lib/imap/sieve
   sendmail: /usr/sbin/sendmail
   hashimapspool: true
   sasl_pwcheck_method: saslauthd
   sasl_mech_list: PLAIN
   tls_cert_file: /usr/share/ssl/certs/crt.crt
   tls_key_file: /usr/share/ssl/certs/key.key
   tls_ca_file: /usr/share/ssl/certs/ca.ca

   allowusermoves: 1

   #Backend servers
   cyrus-be1_password: ***
   cyrus-be2_password: ***
   cyrus-be3_password: ***
   proxy_authname: cyrus


   #Mupdate server
   mupdate_server:10.1.5.101
   mupdate_authname:cyrus
   mupdate_password: ***

   maxmessagesize: 2097152
   syslog_prefix:cyrus
   lmtp_over_quota_perm_failure:1
   quotawarn: 110
   imapidlepoll: 0
   fulldirhash: 1
   munge8bit: 0
   timeout: 10

   tls_session_timeout: 0
   mupdate_connections_max: 1024
   berkeley_cachesize:102400
   berkeley_txns_max:500

#### cyrus versions

Fedora core 2

cyrus-sasl-2.1.18-2.2
cyrus-sasl-md5-2.1.18-2.2
cyrus-sasl-plain-2.1.18-2.2
cyrus-imapd-murder-2.2.10-3.fc2
cyrus-imapd-2.2.10-3.fc2
cyrus-imapd-utils-2.2.10-3.fc2

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html