murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist

[Eric G. Wolfe]

I just wanted to follow up on this thread, rather than leaving it hanging.

It seems there was a more serious issue, which ultimately lead to the 
failure we experienced, with our murder setup.  Specifically our 
internal DNS servers, were having sporadic time-outs for Linux and 
Macintosh clients.  After disabling ipv6 on both of our nameservers, it 
seemed the name resolution issue cleared up.  The strange thing is that 
it has been working without any complications for the last 6 months.  
But nobody at Marshall can pinpoint anything that may have changed 
recently with regards to ipv6 networking on the internal network.  What 
is even stranger is that the name resolution timeouts only seemed to 
occur on Linux and Macintosh clients, while having no harmful effects on 
Microsoft Windows servers and clients.

I personally want to thank everyone who helped troubleshoot and resolve 
this issue.  Wesley Craig at the University of Michigan, thank you for 
your input, steering me in the right direction, and clearing up 
misconceptions about perceived errors.  Andrew Morgan at Oregon State 
University, thank you for the input on working around performance 
bottlenecks (due to a huge backlog of e-mail) during the 
re-synchronization to the front-ends.  Also, a big thanks to Jeff Eaton 
at Carnegie Mellon, for his assistance in troubleshooting and his advice 
on what to do next.

Andrew Morgan wrote:
> On Thu, 20 Nov 2008, Wolfe, Eric G wrote:
>
>   
>> We are using skiplist, I copied the mailboxes.db to frontends.  If the 
>> frontends are updating, it is not apparent.  I could not verify that 
>> either of them were synching from the master after the mupdate master 
>> synch.  Which is why I copied them to the frontends to speed things up.
>>     
>
> I've tried this in the past for the same reasons, but it doesn't work. 
> The only way I've been able to get the correct mailboxes.db on frontends 
> is to let them sync with the mupdate master.  And yes, this takes a while 
> from scratch when you are using skiplist because writes are much slower 
> with skiplist.
>
>   
>> Strange that it would just start causing problems now.  We probably are 
>> seeing a cascading effect of failure, with the backlog though.  Do the 
>> latest vanilla trees, have these patches included in them?  The packages 
>> here: http://cyrusimap.web.cmu.edu/downloads.html#imap.  I am somewhat 
>> reluctant to upgrade things in a fragile state.  If these patches are 
>> included in latest releases, is 2.3.13 a fairly painless upgrade path 
>> from 2.2.12, or do we need to go with 2.2.13?
>>     
>
> I wouldn't upgrade until you can get a working system.
>
>   
>> Is there anything we can turn off in the cyrus.conf or imapd.conf, to 
>> work around this "bottleneck"?  In other words, can we keep the MTA from 
>> knocking on the door for long enough to get everything running smoothly 
>> again?
>>     
>
> Yes, comment out lmtp in your cyrus.conf on the frontends.  After you have 
> successfully synced mailboxes.db, enable lmtp again and restart cyrus on 
> the frontends.  You probably should put some limits on lmtp as well.  We 
> use maxchild=50 here (3 frontends).
>
>   
>> Ok, because it sounds like a problem of connecting to the mupdate master 
>> port on 3905, to the unitiated.
>>     
>
> Hard to say, but take the steps above to simplify the problem.  :)
>
>  	Andy
>   


-- 
Eric G. Wolfe, IT Associate, Sr.
One John Marshall Drive
Marshall University, Drinko Library 428k
Huntington, WV 25755
Phone: 304.696.3428
Email: eric.wolfe at marshall.edu

Nobody's gonna believe that computers are intelligent until they start
coming in late and lying about it.