Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

ellie timoney ellie at fastmail.com
Tue Sep 27 20:08:10 EDT 2016 

Hi Andy,

Bringing this discussion back over here...

> FWIW, I have more empirical data to report on the above server with only
> 2 accounts that are being checked by an old BlackBerry (along with
> Thunderbird an hour or two each day...but the BB is the only device
> checking 24/7)
> 
> It has been ~24 hr since the last cyrus-imapd restart.
> 
> There are now 123 imapd processes, all idled, and all of course for
> those two accounts. Looking at the proc file times in the time-sorted
> list below they span from when the restart happened (Sep 20 0734) to
> just a few minutes ago.
> 
> root at dorcas:~# ls -alt /run/cyrus/proc/
> total 492
> drwx------ 2 cyrus mail 2500 Sep 21 07:19 .
> -rw------- 1 cyrus mail   84 Sep 21 07:10 20204
> [...]

I wonder if we can see what the old processes are actually doing...

Is there any chance you're able to obtain a backtrace of these?

You can use a command something like:

      gdb -p pid /path/to/imapd

where pid is the one of the imapd process ids (which is the last column
in that ls output from /run/cyrus/proc/) and where the correct path to
wherever your imapd executable actually lives (I think this might be
"/usr/lib/cyrus/bin/imapd" on Debian).  You should run this command as
the same user that your cyrus server runs as (probably "cyrus").

Once you're in gdb, look for output like "Reading symbols from
/path/to/imapd...".  If this line also contains "no debugging symbols
found", then we can't get anything useful like this, just "quit" -> "y"
to exit gdb.  But if it does find debugging symbols, great: run "bt" to
obtain a backtrace, then "quit" -> "y" to exit, and send through the
output from the session (probably best to paste the output into a text
file and then attach that to your email).

Another possibility is to trace the system calls.  You can use a command
something like this (also as the "cyrus" user):

    strace -p pid

Do not blindly email the output from this command, it may contain
sensitive/private content!

Let this run for 10 minutes or so, then press ctrl-c to stop it, and
then I guess kind of try to describe the output, and we can go from
there.  If it's very noisy, you've caught it in the middle of doing
something. Wait a little while and try again, or try a different imapd
pid: we're trying to see what the inactive ones are waiting for.

Cheers,

ellie

On Fri, Sep 16, 2016, at 10:40 AM, ellie timoney via Info-cyrus wrote:
> > So, given that all of these report as idle, should they not be shutting
> > down at some point?
>
> "Idle" has special meaning in IMAP.  The last column in the proc/* files
> is the IMAP command currently being executed by the connected client;
> "Idle" here means the client is executing the IDLE command (RFC 2177 if
> you're curious).  It does not mean that the connection is unused and
> could be discarded.
>
> Well-behaved modern clients will spend most of their time in IDLE.
>
> Another interesting thing you can see from the proc/* files is how long
> ago the client's current IMAP command was invoked, by looking at the
> modification time of the proc file (e.g. with ls -l).  You'll probably
> find most of your "Idle" commands were invoked sometime within the last
> half-hour or so, even though the connection itself may have been up for
> many hours.
>
> If you see files in the Idle state that are more than half an hour old,
> then that might indicate a client that has entered the Idle state but
> then lost its connection (which is where the tcp_keepalive settings
> become useful).
>
> > I have the settings below in imapd.conf. My understanding of them is
> > after a connection has been idle for 15 min, try 5 tcp probes at 75 sec
> > intervals before marking the connection as broken.
>
> This is correct (except note that "idle" here takes the general meaning,
> not the IMAP meaning).  But if the connection is still there, then it's
> not broken, and tcp_keepalive won't mark it as broken.
>
> The fact that the connections aren't being disconnected by your
> tcp_keepalive settings indicates that the clients are still alive,
> idling politely until they have something to do (or until the server
> notifies them of activity).
>
> > Also, I see this note in cyrus.conf about idled...
>
> So, the client enters the Idle state when it has nothing useful to do.
> When it wants to execute a new command (maybe because the user has
> clicked something in the interface), it ends the idle state, sends the
> new command and deals with whatever response arrives, etc etc.  When it
> has nothing interesting to do anymore (maybe the user has minimized or
> alt-tabbed away) it re-enters the Idle state.  The client can terminate
> its Idle state and immediately re-enter Idle if it wishes, and will
> typically do so at least every 29 minutes, so that client and server
> both know each other are still there.
>
> During the Idle state, the server sends the client updates if its
> mailboxes change (e.g. due to incoming mail, or due to changes made by a
> different client connected to the same mailbox).  This allows the client
> to get notifications of changes without having to continually ask the
> server "has anything changed yet" -- you can see how this is good
> behaviour.
>
> On the Cyrus side, without "idled", the Idle state is handled by imapd
> periodically checking for activity on the relevant mailboxes.  How often
> it does this is determined by the imapidlepoll setting (default: 60
> seconds).  (This is also the fallback behaviour when there is something
> wrong with idled.)  You can see how a client in this case might not hear
> about activity until up to 60 seconds after it occurs.  Not really
> problematic, but it can feel slow if you (the user) are expecting an
> email to appear any second now (maybe you're concurrently on the phone
> with the sender...).
>
> The idled process, if it's running, allows imapd to notify the client
> instantly when activity occurs, instead of having to
> wait-check-wait-check-wait-check.  It feels a bit faster for the user --
> when they expect an email to arrive, boom there it is.  It's nice, but
> not crucial.
>
> The "imapidlepoll" setting will completely disable the IDLE command if
> it is set to 0 (so don't do that).
>
> The "idlesocket" setting tells both idled and imapd where to create/look
> for the socket used for their communication.  It doesn't need to match
> anywhere else, it just needs to be somewhere sensible that the cyrus
> user has appropriate permissions.  Whatever Debian gives you is probably
> fine.
>
> The idled process just needs to be running to be in use -- imapd will
> use it if it's running and working, and not use it if it's not.  To
> start using it (if you're not already), just enable it in cyrus.conf and
> restart.  It won't make any difference to the number of client
> connections you see.
>
> > # this is only necessary if idlemethod is set to "idled" in imapd.conf
> > idle            cmd="idled"
> >
> > But I can't find anything about idlemethod in any of the cyrus docs or
> > man pages.
>
> ... I have no idea where this "idlemethod" business has come from.  No
> such setting exists in Cyrus, and our example cyrus.conf files all have
> a different comment here:
>
> >  # this is only necessary if using idled for IMAP IDLE
> >  idled                cmd="idled"
>
> I can't find any record of "idlemethod" in commit history either, it's
> not like it's leftover from something that's since changed.  Maybe it's
> from a Debian-specific patch?
>
> Anyway, it sounds like your setup is fine, and things are behaving as
> expected. :)
>
> > I know this user so I will ask her what email client she is using and
> > how often it is set to check email.
>
> How often it is set to check email won't make much difference in this
> case.  That is more for servers that don't support Idle, so the client
> has to keep asking "have you got anything new" every so often.  The fact
> these connections are Idle means it isn't doing that -- it just tells
> the server it's Idle, and then waits for the server to tell it about any
> new activity.
>
> But understanding how the service is actually being used will do a lot
> to calibrate your expectations of how the server should be behaving
> (such as, how many concurrent connections/imapd processes you expect to
> see) :)
>
> Cheers,
>
> ellie
>
> On Thu, Sep 15, 2016, at 08:07 AM, Andy Dorman via Info-cyrus wrote:
> > On 09/13/2016 02:15 AM, Bron Gondwana via Info-cyrus wrote:
> > > On Tue, 13 Sep 2016, at 10:44, ellie timoney via Info-cyrus wrote:
> > >> Note that it talks about the process being used for new connections.
> > >> Each imapd process serves one connection at a time (so if you have 50
> > >> client connections, you will have 50 imapd processes to serve them, plus
> > >> whatever you have preforked ready to serve new connections).  When one
> > >> of these connections disconnects, the imapd process that was serving
> > >> that connection goes back into the pool, ready to serve a new incoming
> > >> connection -- unless it has already served $uses connections, in which
> > >> case it exits, and master spawns a new one instead if necessary.
> > >>
> > >> So you can see how, if you have mostly long-lived connections, each
> > >> imapd's count of how many connections it has served would grow very very
> > >> slowly -- many may not ever reach their limit, because you end up
> > >> needing to reboot the server (OS security updates, data centre works,
> > >> etc) before that occurs.
> > >
> > > If you look in /var/lib/imap/proc/* you should see one file for each process id,
> > > which tells you who is connected to that process, and which folder they have
> > > selected (if any).  This is quite useful for tracking down if there's a single user
> > > causing issues.
> > >
> > > Bron.
> > >
> > >
> >
> > Bron, that was incredibly useful (FWIW, the proc list in Debian is at
> > /run/cyrus/proc/)
> >
> > I took a look and our worst case server appears to have two users
> > (actually one user with two separate addresses) that are causing the
> > trouble. I restarted cyrus-imapd about 10 hrs ago and we are up to 28
> > imapd processes with start times all through the 10-hr block and they
> > all appear to be owned by one of two addresses as you can see here:
> >
> > [snip addresses]
> >
> > So, given that all of these report as idle, should they not be shutting
> > down at some point?
> >
> > I have the settings below in imapd.conf. My understanding of them is
> > after a connection has been idle for 15 min, try 5 tcp probes at 75 sec
> > intervals before marking the connection as broken.
> >
> > tcp_keepalive: 1
> > tcp_keepalive_idle: 900
> > tcp_keepalive_cnt: 5
> > tcp_keepalive_intvl: 75
> >
> > So is it possible that this client is responding to the tcp keepalive
> > packets so the idle connection is never shut down, but the client keeps
> > opening new connections when it wants to check email (which appears to
> > happen about every 10 - 15 minutes)?
> >
> > I know this user so I will ask her what email client she is using and
> > how often it is set to check email.
> >
> > Also, I see this note in cyrus.conf about idled...
> >
> > # this is only necessary if idlemethod is set to "idled" in imapd.conf
> > idle            cmd="idled"
> >
> > But I can't find anything about idlemethod in any of the cyrus docs or
> > man pages.  All I can find to set in imapd.conf for idle is the socket:
> >
> > idlesocket: /var/run/cyrus/socket/idle
> >
> > Hmmmmm...wait a minute...in looking for the Debian proc directory I came
> > across another idle socket created when I restarted cyrus-imapd.
> >
> > srwxrwxrwx 1 root root 0 Sep 14 06:10 /run/cyrus/socket/idle
> >
> > However, our imapd.conf specified a different idle socket which was also
> > created when cyrus-imapd was restarted.
> >
> > srwxrwxrwx 1 root root 0 Sep 14 06:10 /var/run/cyrus/socket/idle
> >
> > So I am thinking I should modify my imapd.conf idlesocket to
> > "/run/cyrus/socket/idle" which cyrus seems to be setting up anyway.  Or
> > am I completely missing an obvious point here?
> >
> > Sorry for being so clueless and thanks for all the help.
> >
> > --
> > Andy Dorman
> > Ironic Design, Inc.
> > AnteSpam.com
> >
> > ----
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

More information about the Info-cyrus mailing list