<html><head></head><body>Have you tried stracing one of the imap processes that is slow to see where the delay is?<br>
<br><br><div class="gmail_quote">On May 19, 2017 6:48:36 AM ADT, Michael Hieb <michael.hieb@celoso.net> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Apologies for long post, most of it is configuration details.<br /><br />I have recently changed from a single threaded cyrus server in which <br />multiple domains were overloaded on one listener as multiple virtual <br />domains to a multiple threaded cyrus server in which multiple domains <br />are listened for on separate ip addresses with separate configurations. <br />The primary motivation was to have separate tls certificates for each <br />domain. The problem is that I can connect to cyrus listener on all <br />ip/ports for all domains, but while one of them will fork immediately <br />and respond with a banner, the others will delay for a period of time <br />from a few seconds to a minute or so before forking and responding with <br />a banner. With the same setup, I change to a single threaded cyrus <br />server to listen on all ip/ports then I get an immediate response on <br />all, but of course I do not get the separate tls certificate or <br />configuration. It is replicable that switching between the single <br />threaded and multiple threaded configuration triggers the problem. I <br />have searched the logs and the mail-lists and found nothing that seems <br />related.<br /><br />Question: why does switching to multiple threaded cyrus server trigger <br />delayed forking of imapd processes after connecting to master listener?<br /><br />On listener which responds immediately (as expected) I get this:<br /><br />user@somehost:~> telnet <a href="http://imap.domain1.com">imap.domain1.com</a> 143<br />Trying <a href="http://192.168.110.171">192.168.110.171</a>...<br />Connected to <a href="http://imap.domain1.com">imap.domain1.com</a>.<br />Escape character is '^]'.<br />* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=GSS-SPNEGO <br />AUTH=GSSAPI AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=LOGIN AUTH=PLAIN SASL-IR] <br /><a href="http://imap.domain1.com">imap.domain1.com</a> Cyrus IMAP v2.4.18 server ready<br /><br />On listener with delay, I get this (and process hangs for many seconds <br />or even a minute or two):<br /><br />user@somehost:~> telnet <a href="http://imap.domain2.com">imap.domain2.com</a> 143<br />Trying <a href="http://192.168.110.171">192.168.110.171</a>...<br />Connected to <a href="http://imap.domain2.com">imap.domain2.com</a>.<br />Escape character is '^]'.<br /><br />The domain for which the listener responds and the one for which it <br />delays seems to change and be random as far as I can tell.<br /><br />There is nothing in the log journal that looks different from when we <br />connect and get a for and when we connect and get a delay.<br /><br />May 19 09:44:31 MAILSERVER master[13762]: about to exec <br />/usr/lib/cyrus/bin/imapd<br />May 19 09:44:31 MAILSERVER imap[13762]: executed<br />May 19 09:44:31 MAILSERVER imap[13762]: IOERROR: opening <br />/var/lib/imap/user_deny.db: No such file or directory<br /><br />and once imapd forks and banner is generated<br /><br />May 19 09:46:45 MAILSERVER imap[13814]: accepted connection<br /><br />Here are the configuration details:<br /><br />I run cyrus 2.4.18-3.6 on openSuSE Leap 42.2 Linux MAILSERVER <br />4.4.62-18.6-default #1 SMP Fri Apr 21 16:14:48 UTC 2017 (84f9824) x86_64 <br />x86_64 x86_64 GNU/Linux.<br /><br />Here is my (sanitized) cyrus.conf and one imapd.conf (they all look <br />alike except for certificate and domain specifics).<br /><br />MAILSERVER:~ # cat /etc/cyrus.conf<br />START {<br /># do not delete this entry!<br />recover cmd="ctl_cyrusdb -r"<br /><br /># this is only necessary if using idled for IMAP IDLE<br />idled cmd="idled"<br />}<br /><br /># UNIX sockets start with a slash and are put into /var/lib/imap/socket<br />SERVICES {<br /># add or remove based on preferences<br />#imap cmd="imapd" listen="imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf " <br />listen="<a href="http://192.168.171.4">192.168.171.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf " <br />listen="<a href="http://192.168.110.171">192.168.110.171</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf " <br />listen="<a href="http://192.168.172.4">192.168.172.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf " <br />listen="<a href="http://192.168.110.172">192.168.110.172</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf " <br />listen="<a href="http://192.168.174.4">192.168.174.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf " <br />listen="<a href="http://192.168.110.174">192.168.110.174</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf " <br />listen="<a href="http://192.168.175.4">192.168.175.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf " <br />listen="<a href="http://192.168.110.175">192.168.110.175</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf " <br />listen="<a href="http://192.168.176.4">192.168.176.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf " <br />listen="<a href="http://192.168.110.176">192.168.110.176</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf " <br />listen="<a href="http://192.168.177.4">192.168.177.4</a>:imap" maxchild=-1 maxforkrate=100<br />imap cmd="imapd -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf " <br />listen="<a href="http://192.168.110.177">192.168.110.177</a>:imap" maxchild=-1 maxforkrate=100<br /><br />#imaps cmd="imapd -s" listen="imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf -s " <br />listen="<a href="http://192.168.171.4">192.168.171.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf -s " <br />listen="<a href="http://192.168.110.171">192.168.110.171</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf -s " <br />listen="<a href="http://192.168.172.4">192.168.172.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf -s " <br />listen="<a href="http://192.168.110.172">192.168.110.172</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf -s " <br />listen="<a href="http://192.168.174.4">192.168.174.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf -s " <br />listen="<a href="http://192.168.110.174">192.168.110.174</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf -s " <br />listen="<a href="http://192.168.175.4">192.168.175.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf -s " <br />listen="<a href="http://192.168.110.175">192.168.110.175</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf -s " <br />listen="<a href="http://192.168.176.4">192.168.176.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf -s " <br />listen="<a href="http://192.168.110.176">192.168.110.176</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf -s " <br />listen="<a href="http://192.168.177.4">192.168.177.4</a>:imaps" maxchild=-1 maxforkrate=100<br />imaps cmd="imapd -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf -s " <br />listen="<a href="http://192.168.110.177">192.168.110.177</a>:imaps" maxchild=-1 maxforkrate=100<br /><br />#pop3 cmd="pop3d" listen="pop3"<br />#pop3s cmd="pop3d -s" listen="pop3s"<br />#sieve cmd="timsieved" listen="sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf" <br />listen="<a href="http://192.168.171.4">192.168.171.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf" <br />listen="<a href="http://192.168.110.171">192.168.110.171</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf" <br />listen="<a href="http://192.168.172.4">192.168.172.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain2.com">imapd.domain2.com</a>.conf" <br />listen="<a href="http://192.168.110.172">192.168.110.172</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf" <br />listen="<a href="http://192.168.174.4">192.168.174.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain3.com">imapd.domain3.com</a>.conf" <br />listen="<a href="http://192.168.110.174">192.168.110.174</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf" <br />listen="<a href="http://192.168.175.4">192.168.175.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain4.com">imapd.domain4.com</a>.conf" <br />listen="<a href="http://192.168.110.175">192.168.110.175</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf" <br />listen="<a href="http://192.168.176.4">192.168.176.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain5.com">imapd.domain5.com</a>.conf" <br />listen="<a href="http://192.168.110.176">192.168.110.176</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf" <br />listen="<a href="http://192.168.177.4">192.168.177.4</a>:sieve" maxchild=-1 maxforkrate=100<br />sieve cmd="timsieved -C /etc/<a href="http://imapd.domain6.com">imapd.domain6.com</a>.conf" <br />listen="<a href="http://192.168.110.177">192.168.110.177</a>:sieve" maxchild=-1 maxforkrate=100<br /><br />#ptloader cmd="ptloader" listen="/var/lib/imap/ptclient/ptsock"<br /><br /># at least one LMTP is required for delivery<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain1.com">mail.domain1.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain2.com">mail.domain2.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain3.com">mail.domain3.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain4.com">mail.domain4.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain5.com">mail.domain5.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />lmtp cmd="lmtpd -a" listen="<a href="http://mail.domain6.com">mail.domain6.com</a>:lmtp" maxchild=-1 <br />maxforkrate=100<br />#lmtpunix cmd="lmtpd" listen="/var/lib/imap/socket/lmtp"<br /><br /># this is only necessary if using notifications<br />#notify cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp"<br />}<br /><br />EVENTS {<br /># this is required<br />checkpoint cmd="ctl_cyrusdb -c" period=30<br /><br /># this is only necessary if using duplicate delivery suppression,<br /># Sieve or NNTP<br />duplicateprune cmd="cyr_expire -E 3" at=0400<br /><br /># Expire data older then 69 days. Two full months of 31 days<br /># each includes two full backup cycles, plus 1 week margin<br /># because we run our full backups on the first sat/sun night<br /># of each month.<br />deleteprune cmd="cyr_expire -E 4 -D 69" at=0430<br />expungeprune cmd="cyr_expire -E 4 -X 69" at=0445<br /><br /># this is only necessary if caching TLS sessions<br />tlsprune cmd="tls_prune" at=0400<br /><br /># Uncomment the next entry, if you want to automatically remove<br /># old messages of EVERY user.<br /># This example calls ipurge every 60 minutes and ipurge will delete<br /># ALL messages older then 120 days.<br /># enter 'man 8 ipurge' for more details<br />#cleanup cmd="ipurge -d 120 -f" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain1.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain2.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain3.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain4.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain5.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain6.com" period=60<br /><br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain1.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain2.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain3.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain4.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain5.com" period=60<br />cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain6.com" period=60<br /><br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain1.com" period=60<br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain2.com" period=60<br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain3.com" period=60<br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain4.com" period=60<br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain5.com" period=60<br />cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain6.com" period=60<br /><br /># Create search indexes regularly<br />squatter cmd="squatter -s -i" at=0530<br /><br /># running sa-learn<br />sa-learn cmd="/usr/local/bin/<a href="http://cyrus-salearn.pl">cyrus-salearn.pl</a>" period=60<br />sa-update cmd="/usr/bin/sa-update -v" at=0000<br /><br />}<br /><br />MAILSERVER:~ # cat /etc/<a href="http://imapd.domain1.com">imapd.domain1.com</a>.conf<br />configdirectory: /var/lib/imap<br />partition-default: /var/spool/imap<br />sievedir: /var/lib/sieve<br />annotation_definitions: /etc/imapd.annotations.conf<br /># admins: cyrus@domain1.net<br />allowanonymouslogin: no<br />autocreatequota: 10000<br />reject8bit: no<br />quotawarn: 90<br />timeout: 30<br />poptimeout: 10<br />dracinterval: 0<br />drachost: localhost<br />sasl_pwcheck_method: saslauthd<br />lmtp_overquota_perm_failure: no<br />#lmtp_catchall_mailbox: admin<br />lmtp_downcase_rcpt: yes<br />lmtp_fuzzy_mailbox_match: yes<br />expunge_mode: delayed<br />deletedprefix: DELETED<br />delete_mode: delayed<br /><br /># added by Michael Hieb Jun 22 2014<br />allowplaintext: yes<br />unixhierarchysep: yes<br />allowplainwithouttls: no<br />altnamespace: no<br />virtdomains: userid<br />servername: <a href="http://imap.domain1.net">imap.domain1.net</a><br /><br />#<br /># if you want TLS, you have to generate certificates and keys<br />#<br />tls_cert_file: /etc/letsencrypt/live/<a href="http://imap.domain1.net/cert.pem">imap.domain1.net/cert.pem</a><br />tls_key_file: /etc/letsencrypt/live/<a href="http://imap.domain1.net/privkey.pem">imap.domain1.net/privkey.pem</a><br />tls_ca_file: /etc/letsencrypt/live/<a href="http://imap.domain1.net/chain.pem">imap.domain1.net/chain.pem</a><br />tls_ca_path: /etc/ssl/certs<br />tls_versions: tls1_0 tls1_1 tls1_2<br /><br /># added by Michael Hieb 20-Jan-2017 for lmtp on port (not socket)<br />lmtp_admins: lmtpuser<br />MAILSERVER:~ #<br /><br />Note: before I discovered the problem with forking could be triggered by <br />switching between single threaded and multi-threaded cyrus server, I <br />thought problem could be related to resource constraints. This is why I <br />set all processes with maxchild=-1 maxforkrate=100. I also set the <br />following in /etc/systemd/system/cyrus.service<br /><br />MAILSERVER:~ # cat /etc/systemd/system/cyrus.service<br />[deleted...]<br />LimitRTPRIO=50000<br />LimitNOFILE=50000<br />LimitNPROC=50000<br />MAILSERVER:~ #<br /><br />None of these resource settings made any difference and the problem goes <br />away as soon as cyrus is single threaded - so I do not believe it is a <br />resource constraint (at least one I understand at this point).<br /><br />Any help would be much appreciated.<br /><br /><br /><br />----<br />Cyrus Home Page: <a href="http://www.cyrusimap.org">http://www.cyrusimap.org</a>/<br />List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus">http://lists.andrew.cmu.edu/pipermail/info-cyrus</a>/<br />To Unsubscribe:<br /><a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br /></pre></blockquote></div></body></html>