Severe problems establishing IMAP connections

[Aidan Evans]

On Fri, 6 Sep 2002 at 12:24 Aidan Evans wrote to info-cyrus at andrew.cmu.edu...

>  We are experiencing severe but inconsistent problems with establishing
>IMAP connections.  If I do a "telnet imap.server 143" it just sits at
>
>	Trying...
>	Connected to imap.UCIS.dal.ca.
>	Escape character is '^]'.
>
>Eventually I may get a response from imapd, but other times I never get
>through.  Curiously, POP connections have similar problems, but much less
>severe; often they get the pop3d banner just about right away, but not
>always.
>
>  If I run netstat and look for the incoming connection it sits in
>SYN_RECV state for minutes.  When the problem is less severe it gets to
>ESTABLISHED, but lsof indicates there is no associated process.
>
>  Could this be a memory shortage?  The machine has two gigabytes.  Right
>now "top" shows
>
>Mem:  2059004K av, 2053712K used,    5292K free,   12K shrd,  207400K buff
>Swap: 2096376K av,   10088K used, 2086288K free              1383220K cached
>
>and there are about 430 imapd processes running plus 370 sendmails.  The
>number of sendmail make it look as though it's also difficult to get lmtpd
>started (via a Unix socket).

  On the weekend Cyrus was updated to version 2.1.9.  Coincidentally the
connections sitting in SYN_RECV state, or ESTABLISHED but no process, hgave
gone away.  However, we still have connection problems.

  What I see now is connections that are established, and have been
accepted by an imapd, but are not getting any response (the IMAP client
says).  Looking at the imapd with strace shows something like

	select(0, NULL, NULL, NULL, [ I've lost the rest of the call ]

being repeated over and over.  The imapd is one that has been reused
several times.  I suspect a simular thing is happening with re-used lmtpd
processes because eventually sendmail-s stop getting connected and begin
to accumulate (over 1,500 at one point).

  On the theory that something is getting confused over the multiple
sessions, I've changed MAX_USE in master/service.h from 250 to 1 in order
to get a new process for each connection, re-compiled, and replaced imapd,
pop3d, and lmtpd.

P.S.  Other things that happened this morning before I restarted Cyrus
include

(a) Many

lmtpd[29636]: DBERROR db4: Unable to allocate memory for transaction detail
lmtpd[29636]: DBERROR: mystore: error beginning txn: Cannot allocate memory

messages.

(b) After I figured out that none of the lmtpd processes were doing
anything I killed them all with TERM signals.  master appeared to recognize
this, but did not start any new ones.

Aidan Evans   | Networks & Systems
(902)494-3332 | University Computing & Information Services
              | Dalhousie University, Halifax, N.S., Canada