Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

[ellie timoney]

> So, given that all of these report as idle, should they not be shutting 
> down at some point?

"Idle" has special meaning in IMAP.  The last column in the proc/* files
is the IMAP command currently being executed by the connected client;
"Idle" here means the client is executing the IDLE command (RFC 2177 if
you're curious).  It does not mean that the connection is unused and
could be discarded.  

Well-behaved modern clients will spend most of their time in IDLE.

Another interesting thing you can see from the proc/* files is how long
ago the client's current IMAP command was invoked, by looking at the
modification time of the proc file (e.g. with ls -l).  You'll probably
find most of your "Idle" commands were invoked sometime within the last
half-hour or so, even though the connection itself may have been up for
many hours.

If you see files in the Idle state that are more than half an hour old,
then that might indicate a client that has entered the Idle state but
then lost its connection (which is where the tcp_keepalive settings
become useful).

> I have the settings below in imapd.conf. My understanding of them is 
> after a connection has been idle for 15 min, try 5 tcp probes at 75 sec 
> intervals before marking the connection as broken.

This is correct (except note that "idle" here takes the general meaning,
not the IMAP meaning).  But if the connection is still there, then it's
not broken, and tcp_keepalive won't mark it as broken.

The fact that the connections aren't being disconnected by your
tcp_keepalive settings indicates that the clients are still alive,
idling politely until they have something to do (or until the server
notifies them of activity).

> Also, I see this note in cyrus.conf about idled...

So, the client enters the Idle state when it has nothing useful to do. 
When it wants to execute a new command (maybe because the user has
clicked something in the interface), it ends the idle state, sends the
new command and deals with whatever response arrives, etc etc.  When it
has nothing interesting to do anymore (maybe the user has minimized or
alt-tabbed away) it re-enters the Idle state.  The client can terminate
its Idle state and immediately re-enter Idle if it wishes, and will
typically do so at least every 29 minutes, so that client and server
both know each other are still there.

During the Idle state, the server sends the client updates if its
mailboxes change (e.g. due to incoming mail, or due to changes made by a
different client connected to the same mailbox).  This allows the client
to get notifications of changes without having to continually ask the
server "has anything changed yet" -- you can see how this is good
behaviour.

On the Cyrus side, without "idled", the Idle state is handled by imapd
periodically checking for activity on the relevant mailboxes.  How often
it does this is determined by the imapidlepoll setting (default: 60
seconds).  (This is also the fallback behaviour when there is something
wrong with idled.)  You can see how a client in this case might not hear
about activity until up to 60 seconds after it occurs.  Not really
problematic, but it can feel slow if you (the user) are expecting an
email to appear any second now (maybe you're concurrently on the phone
with the sender...).

The idled process, if it's running, allows imapd to notify the client
instantly when activity occurs, instead of having to
wait-check-wait-check-wait-check.  It feels a bit faster for the user --
when they expect an email to arrive, boom there it is.  It's nice, but
not crucial.

The "imapidlepoll" setting will completely disable the IDLE command if
it is set to 0 (so don't do that).

The "idlesocket" setting tells both idled and imapd where to create/look
for the socket used for their communication.  It doesn't need to match
anywhere else, it just needs to be somewhere sensible that the cyrus
user has appropriate permissions.  Whatever Debian gives you is probably
fine.

The idled process just needs to be running to be in use -- imapd will
use it if it's running and working, and not use it if it's not.  To
start using it (if you're not already), just enable it in cyrus.conf and
restart.  It won't make any difference to the number of client
connections you see.

> # this is only necessary if idlemethod is set to "idled" in imapd.conf
> idle            cmd="idled"
> 
> But I can't find anything about idlemethod in any of the cyrus docs or 
> man pages.

... I have no idea where this "idlemethod" business has come from.  No
such setting exists in Cyrus, and our example cyrus.conf files all have
a different comment here:

>  # this is only necessary if using idled for IMAP IDLE
>  idled                cmd="idled"

I can't find any record of "idlemethod" in commit history either, it's
not like it's leftover from something that's since changed.  Maybe it's
from a Debian-specific patch?

Anyway, it sounds like your setup is fine, and things are behaving as
expected. :)

> I know this user so I will ask her what email client she is using and 
> how often it is set to check email.

How often it is set to check email won't make much difference in this
case.  That is more for servers that don't support Idle, so the client
has to keep asking "have you got anything new" every so often.  The fact
these connections are Idle means it isn't doing that -- it just tells
the server it's Idle, and then waits for the server to tell it about any
new activity.

But understanding how the service is actually being used will do a lot
to calibrate your expectations of how the server should be behaving
(such as, how many concurrent connections/imapd processes you expect to
see) :)

Cheers,

ellie

On Thu, Sep 15, 2016, at 08:07 AM, Andy Dorman via Info-cyrus wrote:
> On 09/13/2016 02:15 AM, Bron Gondwana via Info-cyrus wrote:
> > On Tue, 13 Sep 2016, at 10:44, ellie timoney via Info-cyrus wrote:
> >> Note that it talks about the process being used for new connections.
> >> Each imapd process serves one connection at a time (so if you have 50
> >> client connections, you will have 50 imapd processes to serve them, plus
> >> whatever you have preforked ready to serve new connections).  When one
> >> of these connections disconnects, the imapd process that was serving
> >> that connection goes back into the pool, ready to serve a new incoming
> >> connection -- unless it has already served $uses connections, in which
> >> case it exits, and master spawns a new one instead if necessary.
> >>
> >> So you can see how, if you have mostly long-lived connections, each
> >> imapd's count of how many connections it has served would grow very very
> >> slowly -- many may not ever reach their limit, because you end up
> >> needing to reboot the server (OS security updates, data centre works,
> >> etc) before that occurs.
> >
> > If you look in /var/lib/imap/proc/* you should see one file for each process id,
> > which tells you who is connected to that process, and which folder they have
> > selected (if any).  This is quite useful for tracking down if there's a single user
> > causing issues.
> >
> > Bron.
> >
> >
> 
> Bron, that was incredibly useful (FWIW, the proc list in Debian is at 
> /run/cyrus/proc/)
> 
> I took a look and our worst case server appears to have two users 
> (actually one user with two separate addresses) that are causing the 
> trouble. I restarted cyrus-imapd about 10 hrs ago and we are up to 28 
> imapd processes with start times all through the 10-hr block and they 
> all appear to be owned by one of two addresses as you can see here:
> 
> [snip addresses]
> 
> So, given that all of these report as idle, should they not be shutting 
> down at some point?
> 
> I have the settings below in imapd.conf. My understanding of them is 
> after a connection has been idle for 15 min, try 5 tcp probes at 75 sec 
> intervals before marking the connection as broken.
> 
> tcp_keepalive: 1
> tcp_keepalive_idle: 900
> tcp_keepalive_cnt: 5
> tcp_keepalive_intvl: 75
> 
> So is it possible that this client is responding to the tcp keepalive 
> packets so the idle connection is never shut down, but the client keeps 
> opening new connections when it wants to check email (which appears to 
> happen about every 10 - 15 minutes)?
> 
> I know this user so I will ask her what email client she is using and 
> how often it is set to check email.
> 
> Also, I see this note in cyrus.conf about idled...
> 
> # this is only necessary if idlemethod is set to "idled" in imapd.conf
> idle            cmd="idled"
> 
> But I can't find anything about idlemethod in any of the cyrus docs or 
> man pages.  All I can find to set in imapd.conf for idle is the socket:
> 
> idlesocket: /var/run/cyrus/socket/idle
> 
> Hmmmmm...wait a minute...in looking for the Debian proc directory I came 
> across another idle socket created when I restarted cyrus-imapd.
> 
> srwxrwxrwx 1 root root 0 Sep 14 06:10 /run/cyrus/socket/idle
> 
> However, our imapd.conf specified a different idle socket which was also 
> created when cyrus-imapd was restarted.
> 
> srwxrwxrwx 1 root root 0 Sep 14 06:10 /var/run/cyrus/socket/idle
> 
> So I am thinking I should modify my imapd.conf idlesocket to 
> "/run/cyrus/socket/idle" which cyrus seems to be setting up anyway.  Or 
> am I completely missing an obvious point here?
> 
> Sorry for being so clueless and thanks for all the help.
> 
> -- 
> Andy Dorman
> Ironic Design, Inc.
> AnteSpam.com
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus