murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist

[Simon Matter]

> I just wanted to follow up on this thread, rather than leaving it hanging.
>
> It seems there was a more serious issue, which ultimately lead to the
> failure we experienced, with our murder setup.  Specifically our
> internal DNS servers, were having sporadic time-outs for Linux and
> Macintosh clients.  After disabling ipv6 on both of our nameservers, it
> seemed the name resolution issue cleared up.  The strange thing is that
> it has been working without any complications for the last 6 months.
> But nobody at Marshall can pinpoint anything that may have changed
> recently with regards to ipv6 networking on the internal network.  What
> is even stranger is that the name resolution timeouts only seemed to
> occur on Linux and Macintosh clients, while having no harmful effects on
> Microsoft Windows servers and clients.

I had a strange issue recently with name resolution (OT because it has
nothing todo with cyrus-imapd).
Resolvers on Linux and recent OSX seem to always try to query for IPv6
(AAAA) records even if NO IPv6 is configured on the client (no, I think it
only happens if the client program tries to resolve IPv6 like firefox in
our case). The problem starts if the DNS servers in question don't reply
correctly to those requests and reply with SERVFAIL for example. The
resolver will then ask the next DNS server it find in resolv.conf and go
on until no more servers to ask, which can take quite some time if you
have 4 servers configured like in our case.
In the end it turned out the company in question were running DNS
loadbalancers and they simply replied with SERVFAIL for AAAA records. I've
been told customers with Windows don't have that problem and I guess it's
because their resolver doesn't ask for IPv6 adresses if no IPv6 is
configured.

Simon

>
> I personally want to thank everyone who helped troubleshoot and resolve
> this issue.  Wesley Craig at the University of Michigan, thank you for
> your input, steering me in the right direction, and clearing up
> misconceptions about perceived errors.  Andrew Morgan at Oregon State
> University, thank you for the input on working around performance
> bottlenecks (due to a huge backlog of e-mail) during the
> re-synchronization to the front-ends.  Also, a big thanks to Jeff Eaton
> at Carnegie Mellon, for his assistance in troubleshooting and his advice
> on what to do next.
>
> Andrew Morgan wrote:
>> On Thu, 20 Nov 2008, Wolfe, Eric G wrote:
>>
>>
>>> We are using skiplist, I copied the mailboxes.db to frontends.  If the
>>> frontends are updating, it is not apparent.  I could not verify that
>>> either of them were synching from the master after the mupdate master
>>> synch.  Which is why I copied them to the frontends to speed things up.
>>>
>>
>> I've tried this in the past for the same reasons, but it doesn't work.
>> The only way I've been able to get the correct mailboxes.db on frontends
>> is to let them sync with the mupdate master.  And yes, this takes a
>> while
>> from scratch when you are using skiplist because writes are much slower
>> with skiplist.
>>
>>
>>> Strange that it would just start causing problems now.  We probably are
>>> seeing a cascading effect of failure, with the backlog though.  Do the
>>> latest vanilla trees, have these patches included in them?  The
>>> packages
>>> here: http://cyrusimap.web.cmu.edu/downloads.html#imap.  I am somewhat
>>> reluctant to upgrade things in a fragile state.  If these patches are
>>> included in latest releases, is 2.3.13 a fairly painless upgrade path
>>> from 2.2.12, or do we need to go with 2.2.13?
>>>
>>
>> I wouldn't upgrade until you can get a working system.
>>
>>
>>> Is there anything we can turn off in the cyrus.conf or imapd.conf, to
>>> work around this "bottleneck"?  In other words, can we keep the MTA
>>> from
>>> knocking on the door for long enough to get everything running smoothly
>>> again?
>>>
>>
>> Yes, comment out lmtp in your cyrus.conf on the frontends.  After you
>> have
>> successfully synced mailboxes.db, enable lmtp again and restart cyrus on
>> the frontends.  You probably should put some limits on lmtp as well.  We
>> use maxchild=50 here (3 frontends).
>>
>>
>>> Ok, because it sounds like a problem of connecting to the mupdate
>>> master
>>> port on 3905, to the unitiated.
>>>
>>
>> Hard to say, but take the steps above to simplify the problem.  :)
>>
>>  	Andy
>>
>
>
> --
> Eric G. Wolfe, IT Associate, Sr.
> One John Marshall Drive
> Marshall University, Drinko Library 428k
> Huntington, WV 25755
> Phone: 304.696.3428
> Email: eric.wolfe at marshall.edu
>
> Nobody's gonna believe that computers are intelligent until they start
> coming in late and lying about it.
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>