DBERROR db3: 17 lockers (was Re: Cyrus IMAPd 2.1.6 Released)

[Aidan Evans]

On Thu, 22 Aug 2002 at 15:37 Simon Josefsson wrote to
info-cyrus at andrew.cmu.edu

>I upgraded to 2.1.7 yesterday, and today I found this in the log:
>
>Aug 22 14:59:34 yxa imapd[14974]: DBERROR db3: 2 lockers
>Aug 22 14:59:48 yxa imapd[14975]: DBERROR db3: 3 lockers
>Aug 22 15:00:02 yxa lmtpd[14995]: DBERROR db3: 4 lockers
>Aug 22 15:00:15 yxa lmtpd[14998]: DBERROR db3: 5 lockers
>Aug 22 15:00:36 yxa ctl_cyrusdb[14999]: checkpointing cyrus databases
>Aug 22 15:00:37 yxa ctl_cyrusdb[14999]: done checkpointing cyrus databases
>Aug 22 15:02:10 yxa lmtpd[15031]: DBERROR db3: 6 lockers
>Aug 22 15:02:35 yxa lmtpd[15040]: DBERROR db3: 7 lockers
>Aug 22 15:03:37 yxa imapd[14414]: DBERROR db3: 8 lockers
>Aug 22 15:04:09 yxa lmtpd[15060]: DBERROR db3: 9 lockers
>
>and the number of lockers soon escalated.  During this time, local
>mail delivery stopped (may have been only one user that was stopped
>though).
>
>I was reading mail when this started, perhaps some kind of race
>condition.
>
>Restarting master solved it.
>
>Anyone else seen this?

  I was hoping that the problem you describe was the same as the similar
one we've been having today.  The external symptoms we experience are very,
very long delays in connections being accepted, very large numbers of
sendmail and scores of lmtpd processes building up, the "mbpath" command
taking as much as 30 seconds to run.  In imapd.log the numbers of "DBERROR
... lockers" messages grow from a few a minute to over a thousand a minute.

  It was suggested in a reply to your original message that mailboxes.db be
done with skiplist, so I shutdown Cyrus and switched mailboxes.db from
Berkeley 4 to skiplist.  This may have helped somewhat, but I'm still
accumulating thousands of sendmails and 252 lmptds.  Also I'm seeing
numerous "DBERROR:  mystore: error beginning txn:  Cannot allocate memory"
messages from lmtpd that did not occur before I switched the mailboxes.db
database.  These stopped after I stopped sendmail accepting new connections
and let all the running sendmails complete.  But now lmtpd is ignoring
connection attempts.  I killed the last one running and no more have
started, so mail delivery is dead.

  Time to restart master again I guess.  Any ideas about the cause of these
difficulties?

Name       : Cyrus IMAPD
version    : v2.1.4 2002/04/30 18:06:05
vendor     : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os         : Linux
os-version : 2.4.9-34smp
environment: Cyrus SASL 2.1.2
             Sleepycat Software: Berkeley DB 4.0.14: (November 18, 2001)
             OpenSSL 0.9.6b [engine] 9 Jul 2001
             CMU Sieve 2.1
             TCP Wrappers
             lock = fcntl
             auth = unix
             idle = poll
             mboxlist.db = skiplist
             subs.db = flat
             seen.db = flat
             duplicate.db = db3-nosync
             tls.db = db3-nosync

Aidan Evans   | Networks & Systems
(902)494-3332 | University Computing & Information Services
              | Dalhousie University, Halifax, N.S., Canada