murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist

[Eric G. Wolfe]

We have a RHEL4u7 on all 5 servers:
1 mupdate master: mumailmaster
2 backends: mumailstore01, mumailstore02
2 Postfix MTA/Cyrus proxy frontends: mumail01, mumail02

So I started getting this on my backends around 14:15 EST, at which time 
mail started getting deferred to the backends.  I have 20,000+ per 
frontend deferred for delivery at the time of this e-mail.

Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR db4: PANIC: 
Cannot allocate memory
Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4672]: DBERROR: critical 
database situation
Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR db4: PANIC: 
fatal region error detected; run recovery
Nov 19 16:30:26 mumailstore01 ctl_mboxlist[4673]: DBERROR: critical 
database situation
Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR db4: PANIC: 
fatal region error detected; run recovery
Nov 19 16:30:26 mumailstore01 ctl_cyrusdb[4674]: DBERROR: critical 
database situation

I tried the following directions for recovery. 
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&searchterm=skiplist&msg=32337.  
I made backup copies of all files deleted, renaming them 
$filename.corrupt.  I did this on each server, recovered on the 
backends, and let it push the updates to the mupdate server.  Manually 
recovered mailboxes.db on the frontends, as they did not seem to be 
getting updated.  If I am going about this wrong, please someone point 
me in the right direction for documentation on murder disaster recovery. 
The following, while somewhat helpful, does not go into a great amount 
of detail: http://cyrusimap.web.cmu.edu/imapd/install-murder.html.

So at this point my user agent says Unknown/Invalid partition.  The 
partitions are correctly defined on the backend mail stores.  A 
'ctl_mboxlist -d' shows correct partitions, no matter which local 
mailboxes.db I attempt to dump.  Furthermore, LMTP is still not 
delivering to the backends during this outage.

Any helpful tips or pointers would be appreciated.

Thanks,

-- 
Eric G. Wolfe, IT Associate, Sr.
One John Marshall Drive
Marshall University, Drinko Library 428k
Huntington, WV 25755
Phone: 304.696.3428
Email: eric.wolfe at marshall.edu

"Who is General Failure and why is he reading my hard disk ?"
Microsoft spel chekar vor sail, worgs grate !!
(By leitner at inf.fu-berlin.de, Felix von Leitner)