Cyrus 2.5.9 imapd children and tcp_timeout questions
ellie timoney
ellie at fastmail.com
Mon Sep 12 20:44:25 EDT 2016
> We do not have a lot of active users on each server (probably under 20),
> but enough that what you describe could be happening.
Keep in mind too, that your users might have multiple clients running at
the same time -- a desktop that's always on, a laptop that comes and
goes when the laptop is in use, their phone/tablet/etc -- each of which
might use multiple connections like Thunderbird does. You might see as
many as 15 legitimate concurrent connections (and thus imapd processes)
for a single user account in normal use in this particular use case.
> As for how I am determining they are reaching their usage limit, all I
> have are the process start times. When we hit the 100 limit I can see
> that the preforked 5 are still running as are many that were started
> many hours ago.
I would expect exactly this if you have long-lived client connections,
which is normal for IMAP.
If those first five clients (which might really be five connections from
a single client) don't disconnect themselves, and their network
connection doesn't drop out, and they behave well with respect to
IDLE/NOOP, and the server doesn't get restarted or lose its network
connection, then those connections, and therefore those particular
processes, will keep going. This is how it's supposed to work. :)
> I could be crazy wrong (probably am ;-) but it seemed to
> me that if processes were being killed and recycled I should not see a
> lot of old processes sitting around.
Here's the section of imapd(8) describing the uses limit:
> -U uses
> The maximum number of times that the process should be used for new connections before shutting down. The default is 250.
Note that it talks about the process being used for new connections.
Each imapd process serves one connection at a time (so if you have 50
client connections, you will have 50 imapd processes to serve them, plus
whatever you have preforked ready to serve new connections). When one
of these connections disconnects, the imapd process that was serving
that connection goes back into the pool, ready to serve a new incoming
connection -- unless it has already served $uses connections, in which
case it exits, and master spawns a new one instead if necessary.
So you can see how, if you have mostly long-lived connections, each
imapd's count of how many connections it has served would grow very very
slowly -- many may not ever reach their limit, because you end up
needing to reboot the server (OS security updates, data centre works,
etc) before that occurs.
Cheers,
ellie
On Mon, Sep 12, 2016, at 10:40 PM, Andy Dorman via Info-cyrus wrote:
> Eliie, thank you for that information about imapd usage. I have only
> been managing our imap for a few months and am still learning.
>
> We do not have a lot of active users on each server (probably under 20),
> but enough that what you describe could be happening.
>
> As for how I am determining they are reaching their usage limit, all I
> have are the process start times. When we hit the 100 limit I can see
> that the preforked 5 are still running as are many that were started
> many hours ago. I could be crazy wrong (probably am ;-) but it seemed to
> me that if processes were being killed and recycled I should not see a
> lot of old processes sitting around.
>
> And I owe an apology to Konrad...I originally wrote this email a week
> ago and then set it aside while I tried setting tcp_timeout=1. Then
> when that did not help, I sent the email and forgot to update my
> imapd.conf. This is what I have in my imapd.conf.
>
> tcp_keepalive: 1
> tcp_keepalive_idle: 900
> tcp_keepalive_cnt: 5
> tcp_keepalive_intvl: 75
>
> It would be great if anyone who is using tcp_keepalive could share their
> settings in case I completely misunderstood how these work.
>
> In the meantime I am going to bump my max processes up to 200 and my
> monitoring warning level to 150 see how that works.
>
> Thanks for all the help everyone.
>
> Andy
>
> On 09/11/2016 11:05 PM, ellie timoney via Info-cyrus wrote:
> >> We now have 4 of the 14 servers where the number of imapd processes
> >> slowly grows and usually within 12-14 hours reaches the imapd process
> >> limit (currently set to 100).
> >
> > How many user accounts are on these servers? Each client connection
> > constitutes one imapd process on the server.
> >
> > It's my understanding that Thunderbird (for example) in its default
> > configuration holds 5 connections open at once, and tries to keep them
> > open (either with IDLE, if enabled, or NOOP), and will reconnect them
> > after a while if they disconnect. (I would expect most other IMAP
> > clients of a similar maturity to behave similarly.) So it would only
> > take 20 users leaving their mail client open most of the time to hit
> > your limit of 100 concurrent imapd processes.
> >
> > So 100 seems like a very small number, unless you have a very small
> > number of user accounts.
> >
> >> From what we can tell watching the
> >> process list, the old processes are never shut down after reaching their
> >> usage limit (which is the default).
> >
> > Do you have more information about how you determined that their usage
> > limit had been reached?
> >
> >> While we had this problem intermittently with 2.4.18 (once or twice a
> >> month on a random server), it is much worse since the update to 2.5.9.
> >
> > This might simply be 2.5.9 doing a better job of helping clients keep
> > open connections open?
> >
> > Cheers,
> >
> > ellie
> >
> > On Mon, Sep 12, 2016, at 01:01 PM, Andy Dorman via Info-cyrus wrote:
> >> Hi everyone. We have 14 Debian servers running the current Debian
> >> release of Cyrus imap 2.5.9.
> >>
> >> We upgraded from 2.4.18 to 2.5.9 on Aug 28, upgraded the cyrus.conf &
> >> imapd.conf to rename the deprecated options and switch from skiplist to
> >> twoskip for all our dbs. Our cyrus.conf and imapd.con (minus comments)
> >> is at the end of this email.
> >>
> >> We now have 4 of the 14 servers where the number of imapd processes
> >> slowly grows and usually within 12-14 hours reaches the imapd process
> >> limit (currently set to 100). From what we can tell watching the
> >> process list, the old processes are never shut down after reaching their
> >> usage limit (which is the default).
> >>
> >> The difference between the 4 with the problem and the others have to be
> >> the mail clients. Each server supports a different group of mail
> >> domains and users, so I suspect some of the users on the problem servers
> >> are using one or more mail clients that are not well behaved, but I have
> >> no indication of what they are doing wrong.
> >>
> >> While we had this problem intermittently with 2.4.18 (once or twice a
> >> month on a random server), it is much worse since the update to 2.5.9.
> >> We now have to restart imapd on all the problem servers almost twice a
> >> day.
> >>
> >> If it matters, we have tried both idled and polling and have not seen
> >> any difference.
> >>
> >> So has anyone else seen an issue like this? Does anyone have any
> >> suggestions on how to debug?
> >>
> >> We have examined syslog and do not see any strange reports outside of
> >> the normal berkeley DBERROR messages we always see due to bdb compiled
> >> into the debian Cyrus release.
> >>
> >> ===== cyrus.conf =====
> >> START {
> >> recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r"
> >> idle cmd="idled"
> >> delprune cmd="/usr/sbin/cyrus expire -E 3"
> >> tlsprune cmd="/usr/sbin/cyrus tls_prune"
> >> }
> >>
> >> SERVICES {
> >> imap cmd="imapd" listen="*:imap" prefork=5 maxchild=100
> >> imaps cmd="imapd -s" listen="*:imaps" prefork=5 maxchild=100
> >> lmtp cmd="lmtpd" listen="*:lmtp" prefork=5 maxchild=20
> >> sieve cmd="timsieved" listen="*:sieve" prefork=0 maxchild=100
> >> }
> >>
> >> EVENTS {
> >> checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30
> >> delprune cmd="/usr/sbin/cyrus expire -E 3" period=120
> >> tlsprune cmd="/usr/sbin/cyrus tls_prune" period=60
> >> resquat cmd="/usr/sbin/cyrus squatter -s -r -i *" at=0437
> >> }
> >>
> >> ===== imapd.conf =====
> >> admins: cyrus
> >>
> >> allowallsubscribe: on
> >>
> >> allowanonymouslogin: no
> >>
> >> allowplaintext: on
> >>
> >> altnamespace: on
> >>
> >> annotation_db: twoskip
> >>
> >> anyoneuseracl: off
> >>
> >> autocreate_quota: 512000
> >>
> >> configdirectory: /var/lib/cyrus
> >>
> >> defaultdomain: ironicdesign.com
> >>
> >> defaultpartition: default
> >>
> >> duplicate_db: twoskip
> >>
> >> expunge_mode: delayed
> >>
> >> fulldirhash: on
> >>
> >> guid_mode: sha1
> >>
> >> hashimapspool: on
> >>
> >> idlesocket: /var/run/cyrus/socket/idle
> >>
> >> improved_mboxlist_sort: on
> >>
> >> lmtp_downcase_rcpt: on
> >>
> >> lmtpsocket: /var/run/cyrus/socket/lmtp
> >>
> >> mboxname_lockpath: /run/cyrus/lock
> >>
> >> notifysocket: /var/run/cyrus/socket/notify
> >>
> >> partition-default: /var/spool/cyrus/mail
> >>
> >> popminpoll: 1
> >>
> >> proc_path: /run/cyrus/proc
> >>
> >> quota_db: twoskip
> >>
> >> sasl_log_level: 7
> >> sasl_mech_list: PLAIN
> >> sasl_pwcheck_method: saslauthd
> >> sasl_minimum_layer: 0
> >> allowapop: no
> >>
> >> sieveusehomedir: false
> >> sievedir: /var/spool/cyrus/sieve
> >>
> >> statuscache_db: twoskip
> >>
> >> syslog_prefix: cyrus
> >>
> >> tls_sessions_db: twoskip
> >>
> >> tls_client_ca_file: /etc/ssl/certs/mail.ironicdesign.com.pem
> >>
> >> tls_client_ca_dir: /etc/ssl/certs
> >>
> >> # The rest of our TLS setup
> >> tls_server_cert: /etc/ssl/certs/mail.ironicdesign.com.pem
> >> tls_server_key: /etc/ssl/certs/mail.ironicdesign.com.pem
> >> tls_session_timeout: 1440
> >>
> >> tls_ciphers: TLSv1+HIGH:!aNULL:@STRENGTH
> >>
> >> tls_versions: tls1_0 tls1_1 tls1_2
> >>
> >> umask: 077
> >>
> >> unix_group_enable: off
> >>
> >> unixhierarchysep: on
> >>
> >> userdeny_db: twoskip
> >>
> >> virtdomains: userid
> >>
> >> sieve_extensions: fileinto reject vacation imapflags notify envelope
> >> body relational regex subaddress copy
> >> ===== end imapd.conf =====
> >>
> >> Thanks for any ideas on what to look for or how to debug this.
> >>
> >> --
> >> Andy Dorman
> >> Ironic Design, Inc.
> >> AnteSpam.com
> >>
> >> ----
> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> To Unsubscribe:
> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > ----
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
More information about the Info-cyrus
mailing list