pointless select()s and IMAP connections in CLOSE_WAIT

[Michael Wood]

Hi

We're running Cyrus imapd version 2.1.17 on a FreeBSD server
(4.11-RELEASE) with Cyrus SASL 2.1.19 and saslauthd.  Everything
is installed from the ports tree.  In the middle of the day we
get up to about 2000 concurrent connections to the IMAP port.

Usually these connections are all ESTABLISHED according to
netstat/lsof etc., but sometimes we suddenly get a huge spike in
connections.  If I look at netstat at that point, all the
"extra" connections are in the CLOSE_WAIT state.

# netstat -n | grep CLOSE_WAIT
[...]
tcp4     102      0  xxx.yyy.zzz.www.993  xxx.yyy.65.101.1480   CLOSE_WAIT
tcp4     105      0  xxx.yyy.zzz.www.993  xxx.yyy.5.211.4289    CLOSE_WAIT
tcp4       0      0  xxx.yyy.zzz.www.993  xxx.yyy.35.232.1721   CLOSE_WAIT
tcp4      84      0  xxx.yyy.zzz.www.993  xxx.yyy.28.3.3480     CLOSE_WAIT
tcp4      98      0  xxx.yyy.zzz.www.993  xxx.yyy.66.14.50285   CLOSE_WAIT
tcp4       0      0  xxx.yyy.zzz.www.993  xxx.yyy.28.34.1373    CLOSE_WAIT
tcp4       0      0  xxx.yyy.zzz.www.993  xxx.yyy.35.232.3552   CLOSE_WAIT
tcp4      57  32147  xxx.yyy.zzz.www.993  xxx.yyy.5.182.1316    CLOSE_WAIT

The only fix for the problem appears to be to restart Cyrus.

It I have a look at one of the processes with lsof, this is what
it looks like:

# lsof -i @xxx.yyy.65.101:1480
COMMAND   PID  USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
imapd   85462 cyrus    0u  IPv4 0xef4a0c00      0t0  TCP xxx.yyy.zzz.www:imaps->xxx.yyy.65.101:1480 (CLOSE_WAIT)
imapd   85462 cyrus    1u  IPv4 0xef4a0c00      0t0  TCP xxx.yyy.zzz.www:imaps->xxx.yyy.65.101:1480 (CLOSE_WAIT)
imapd   85462 cyrus    2u  IPv4 0xef4a0c00      0t0  TCP xxx.yyy.zzz.www:imaps->xxx.yyy.65.101:1480 (CLOSE_WAIT)

If I look at the process with truss, it seems to be stuck in a
loop calling select with no file descriptors!

# truss -p 85462
(null)()                                         = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)
select(0x0,0x0,0x0,0x0,0xbfbfe808)               = 0 (0x0)

It seems as if some of the imapds suddenly forget which fds they
are supposed to be reading from/writing to and just wait around
for something to happen that never will.

I've had a look around on the Cyrus mailing list archives, but
haven't managed to find a solution to this problem.  Is it a
known problem?

Thanks.

-- 
Michael Wood <mwood at icts.uct.ac.za>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html