<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Hi Andy,<br></div>
<div><br></div>
<div>I remember writing a faq about process counts growing. </div>
<div><br></div>
<div><a href="https://cyrusimap.org/imap/faqs/o-toomanyprocesses.html" class="">https://cyrusimap.org/imap/faqs/o-toomanyprocesses.html</a><br></div>
<div><br></div>
<div>I'm not actually sure it's applicable in your situation, but just in case...<br></div>
<div><br></div>
<div>Cheers,<br></div>
<div><br></div>
<div> Nicola<br></div>
<div><br></div>
<div><br></div>
<div>----- Original message -----<br></div>
<div>From: "Andy Dorman via Info-cyrus" <info-cyrus@lists.andrew.cmu.edu><br></div>
<div>To: info-cyrus@lists.andrew.cmu.edu<br></div>
<div>Subject: Re: Cyrus 2.5.9 imapd children and tcp_timeout questions<br></div>
<div>Date: Mon, 12 Sep 2016 07:40:28 -0500<br></div>
<div><br></div>
<div>Eliie, thank you for that information about imapd usage. I have only<br></div>
<div>been managing our imap for a few months and am still learning.<br></div>
<div><br></div>
<div>We do not have a lot of active users on each server (probably under 20),<br></div>
<div>but enough that what you describe could be happening.<br></div>
<div><br></div>
<div>As for how I am determining they are reaching their usage limit, all I<br></div>
<div>have are the process start times. When we hit the 100 limit I can see<br></div>
<div>that the preforked 5 are still running as are many that were started<br></div>
<div>many hours ago. I could be crazy wrong (probably am ;-) but it seemed to<br></div>
<div>me that if processes were being killed and recycled I should not see a<br></div>
<div>lot of old processes sitting around.<br></div>
<div><br></div>
<div>And I owe an apology to Konrad...I originally wrote this email a week<br></div>
<div>ago and then set it aside while I tried setting tcp_timeout=1. Then<br></div>
<div>when that did not help, I sent the email and forgot to update my<br></div>
<div>imapd.conf. This is what I have in my imapd.conf.<br></div>
<div><br></div>
<div>tcp_keepalive: 1<br></div>
<div>tcp_keepalive_idle: 900<br></div>
<div>tcp_keepalive_cnt: 5<br></div>
<div>tcp_keepalive_intvl: 75<br></div>
<div><br></div>
<div>It would be great if anyone who is using tcp_keepalive could share their<br></div>
<div>settings in case I completely misunderstood how these work.<br></div>
<div><br></div>
<div>In the meantime I am going to bump my max processes up to 200 and my<br></div>
<div>monitoring warning level to 150 see how that works.<br></div>
<div><br></div>
<div>Thanks for all the help everyone.<br></div>
<div><br></div>
<div>Andy<br></div>
<div><br></div>
<div>On 09/11/2016 11:05 PM, ellie timoney via Info-cyrus wrote:<br></div>
<div>>> We now have 4 of the 14 servers where the number of imapd processes<br></div>
<div>>> slowly grows and usually within 12-14 hours reaches the imapd process<br></div>
<div>>> limit (currently set to 100).<br></div>
<div>><br></div>
<div>> How many user accounts are on these servers? Each client connection<br></div>
<div>> constitutes one imapd process on the server.<br></div>
<div>><br></div>
<div>> It's my understanding that Thunderbird (for example) in its default<br></div>
<div>> configuration holds 5 connections open at once, and tries to keep them<br></div>
<div>> open (either with IDLE, if enabled, or NOOP), and will reconnect them<br></div>
<div>> after a while if they disconnect. (I would expect most other IMAP<br></div>
<div>> clients of a similar maturity to behave similarly.) So it would only<br></div>
<div>> take 20 users leaving their mail client open most of the time to hit<br></div>
<div>> your limit of 100 concurrent imapd processes.<br></div>
<div>><br></div>
<div>> So 100 seems like a very small number, unless you have a very small<br></div>
<div>> number of user accounts.<br></div>
<div>><br></div>
<div>>> From what we can tell watching the<br></div>
<div>>> process list, the old processes are never shut down after reaching their<br></div>
<div>>> usage limit (which is the default).<br></div>
<div>><br></div>
<div>> Do you have more information about how you determined that their usage<br></div>
<div>> limit had been reached?<br></div>
<div>><br></div>
<div>>> While we had this problem intermittently with 2.4.18 (once or twice a<br></div>
<div>>> month on a random server), it is much worse since the update to 2.5.9.<br></div>
<div>><br></div>
<div>> This might simply be 2.5.9 doing a better job of helping clients keep<br></div>
<div>> open connections open?<br></div>
<div>><br></div>
<div>> Cheers,<br></div>
<div>><br></div>
<div>> ellie<br></div>
<div>><br></div>
<div>> On Mon, Sep 12, 2016, at 01:01 PM, Andy Dorman via Info-cyrus wrote:<br></div>
<div>>> Hi everyone. We have 14 Debian servers running the current Debian<br></div>
<div>>> release of Cyrus imap 2.5.9.<br></div>
<div>>><br></div>
<div>>> We upgraded from 2.4.18 to 2.5.9 on Aug 28, upgraded the cyrus.conf &<br></div>
<div>>> imapd.conf to rename the deprecated options and switch from skiplist to<br></div>
<div>>> twoskip for all our dbs. Our cyrus.conf and imapd.con (minus comments)<br></div>
<div>>> is at the end of this email.<br></div>
<div>>><br></div>
<div>>> We now have 4 of the 14 servers where the number of imapd processes<br></div>
<div>>> slowly grows and usually within 12-14 hours reaches the imapd process<br></div>
<div>>> limit (currently set to 100). From what we can tell watching the<br></div>
<div>>> process list, the old processes are never shut down after reaching their<br></div>
<div>>> usage limit (which is the default).<br></div>
<div>>><br></div>
<div>>> The difference between the 4 with the problem and the others have to be<br></div>
<div>>> the mail clients. Each server supports a different group of mail<br></div>
<div>>> domains and users, so I suspect some of the users on the problem servers<br></div>
<div>>> are using one or more mail clients that are not well behaved, but I have<br></div>
<div>>> no indication of what they are doing wrong.<br></div>
<div>>><br></div>
<div>>> While we had this problem intermittently with 2.4.18 (once or twice a<br></div>
<div>>> month on a random server), it is much worse since the update to 2.5.9.<br></div>
<div>>> We now have to restart imapd on all the problem servers almost twice a<br></div>
<div>>> day.<br></div>
<div>>><br></div>
<div>>> If it matters, we have tried both idled and polling and have not seen<br></div>
<div>>> any difference.<br></div>
<div>>><br></div>
<div>>> So has anyone else seen an issue like this? Does anyone have any<br></div>
<div>>> suggestions on how to debug?<br></div>
<div>>><br></div>
<div>>> We have examined syslog and do not see any strange reports outside of<br></div>
<div>>> the normal berkeley DBERROR messages we always see due to bdb compiled<br></div>
<div>>> into the debian Cyrus release.<br></div>
<div>>><br></div>
<div>>> ===== cyrus.conf =====<br></div>
<div>>> START {<br></div>
<div>>> recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r"<br></div>
<div>>> idle cmd="idled"<br></div>
<div>>> delprune cmd="/usr/sbin/cyrus expire -E 3"<br></div>
<div>>> tlsprune cmd="/usr/sbin/cyrus tls_prune"<br></div>
<div>>> }<br></div>
<div>>><br></div>
<div>>> SERVICES {<br></div>
<div>>> imap cmd="imapd" listen="*:imap" prefork=5 maxchild=100<br></div>
<div>>> imaps cmd="imapd -s" listen="*:imaps" prefork=5 maxchild=100<br></div>
<div>>> lmtp cmd="lmtpd" listen="*:lmtp" prefork=5 maxchild=20<br></div>
<div>>> sieve cmd="timsieved" listen="*:sieve" prefork=0 maxchild=100<br></div>
<div>>> }<br></div>
<div>>><br></div>
<div>>> EVENTS {<br></div>
<div>>> checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30<br></div>
<div>>> delprune cmd="/usr/sbin/cyrus expire -E 3" period=120<br></div>
<div>>> tlsprune cmd="/usr/sbin/cyrus tls_prune" period=60<br></div>
<div>>> resquat cmd="/usr/sbin/cyrus squatter -s -r -i *" at=0437<br></div>
<div>>> }<br></div>
<div>>><br></div>
<div>>> ===== imapd.conf =====<br></div>
<div>>> admins: cyrus<br></div>
<div>>><br></div>
<div>>> allowallsubscribe: on<br></div>
<div>>><br></div>
<div>>> allowanonymouslogin: no<br></div>
<div>>><br></div>
<div>>> allowplaintext: on<br></div>
<div>>><br></div>
<div>>> altnamespace: on<br></div>
<div>>><br></div>
<div>>> annotation_db: twoskip<br></div>
<div>>><br></div>
<div>>> anyoneuseracl: off<br></div>
<div>>><br></div>
<div>>> autocreate_quota: 512000<br></div>
<div>>><br></div>
<div>>> configdirectory: /var/lib/cyrus<br></div>
<div>>><br></div>
<div>>> defaultdomain: ironicdesign.com<br></div>
<div>>><br></div>
<div>>> defaultpartition: default<br></div>
<div>>><br></div>
<div>>> duplicate_db: twoskip<br></div>
<div>>><br></div>
<div>>> expunge_mode: delayed<br></div>
<div>>><br></div>
<div>>> fulldirhash: on<br></div>
<div>>><br></div>
<div>>> guid_mode: sha1<br></div>
<div>>><br></div>
<div>>> hashimapspool: on<br></div>
<div>>><br></div>
<div>>> idlesocket: /var/run/cyrus/socket/idle<br></div>
<div>>><br></div>
<div>>> improved_mboxlist_sort: on<br></div>
<div>>><br></div>
<div>>> lmtp_downcase_rcpt: on<br></div>
<div>>><br></div>
<div>>> lmtpsocket: /var/run/cyrus/socket/lmtp<br></div>
<div>>><br></div>
<div>>> mboxname_lockpath: /run/cyrus/lock<br></div>
<div>>><br></div>
<div>>> notifysocket: /var/run/cyrus/socket/notify<br></div>
<div>>><br></div>
<div>>> partition-default: /var/spool/cyrus/mail<br></div>
<div>>><br></div>
<div>>> popminpoll: 1<br></div>
<div>>><br></div>
<div>>> proc_path: /run/cyrus/proc<br></div>
<div>>><br></div>
<div>>> quota_db: twoskip<br></div>
<div>>><br></div>
<div>>> sasl_log_level: 7<br></div>
<div>>> sasl_mech_list: PLAIN<br></div>
<div>>> sasl_pwcheck_method: saslauthd<br></div>
<div>>> sasl_minimum_layer: 0<br></div>
<div>>> allowapop: no<br></div>
<div>>><br></div>
<div>>> sieveusehomedir: false<br></div>
<div>>> sievedir: /var/spool/cyrus/sieve<br></div>
<div>>><br></div>
<div>>> statuscache_db: twoskip<br></div>
<div>>><br></div>
<div>>> syslog_prefix: cyrus<br></div>
<div>>><br></div>
<div>>> tls_sessions_db: twoskip<br></div>
<div>>><br></div>
<div>>> tls_client_ca_file: /etc/ssl/certs/mail.ironicdesign.com.pem<br></div>
<div>>><br></div>
<div>>> tls_client_ca_dir: /etc/ssl/certs<br></div>
<div>>><br></div>
<div>>> # The rest of our TLS setup<br></div>
<div>>> tls_server_cert: /etc/ssl/certs/mail.ironicdesign.com.pem<br></div>
<div>>> tls_server_key: /etc/ssl/certs/mail.ironicdesign.com.pem<br></div>
<div>>> tls_session_timeout: 1440<br></div>
<div>>><br></div>
<div>>> tls_ciphers: TLSv1+HIGH:!aNULL:@STRENGTH<br></div>
<div>>><br></div>
<div>>> tls_versions: tls1_0 tls1_1 tls1_2<br></div>
<div>>><br></div>
<div>>> umask: 077<br></div>
<div>>><br></div>
<div>>> unix_group_enable: off<br></div>
<div>>><br></div>
<div>>> unixhierarchysep: on<br></div>
<div>>><br></div>
<div>>> userdeny_db: twoskip<br></div>
<div>>><br></div>
<div>>> virtdomains: userid<br></div>
<div>>><br></div>
<div>>> sieve_extensions: fileinto reject vacation imapflags notify envelope<br></div>
<div>>> body relational regex subaddress copy<br></div>
<div>>> ===== end imapd.conf =====<br></div>
<div>>><br></div>
<div>>> Thanks for any ideas on what to look for or how to debug this.<br></div>
<div>>><br></div>
<div>>> --<br></div>
<div>>> Andy Dorman<br></div>
<div>>> Ironic Design, Inc.<br></div>
<div>>> AnteSpam.com<br></div>
<div>>><br></div>
<div>>> ----<br></div>
<div>>> Cyrus Home Page: http://www.cyrusimap.org/<br></div>
<div>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/<br></div>
<div>>> To Unsubscribe:<br></div>
<div>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus<br></div>
<div>> ----<br></div>
<div>> Cyrus Home Page: http://www.cyrusimap.org/<br></div>
<div>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/<br></div>
<div>> To Unsubscribe:<br></div>
<div>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus<br></div>
<div>><br></div>
<div><br></div>
<div>----<br></div>
<div>Cyrus Home Page: http://www.cyrusimap.org/<br></div>
<div>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/<br></div>
<div>To Unsubscribe:<br></div>
<div>https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus<br></div>
<div><br></div>
</body>
</html>