cyrus master fails with status 71

[Bron Gondwana]

On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:
> Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2 
> (release-p7) ports tree.
> 
> The cyrus master process is failing periodically (every 1-2 weeks) as 
> follows:
> 
> Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps 
> path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71
> Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in 
> READY state: terminated abnormally
> Oct 22 07:38:48 imap1 master[7767]: too many failures for service 
> imaps/ipv4, disabling until next SIGHUP
> 
> This prevents new connections by clients until cyrus is restarted.  I've 
> looked around the web but have not seen this issue reported.
> 
> A little background:
> 
> Our initial thought on this was that we were running out of listen 
> queues so have upped that incrementally from the default of 32 to a 
> current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l 
> option, with increased kern.ipc.soacceptqueue set to 32768, but that 
> hasn't helped.  Sometimes the "status 71" occurs during periods of light 
> use during off hours, like on Saturday mornings.
> 
> We have ~1400 imap accounts, though the number of impad processes hovers 
> around 3,000-4,000.  There have been spikes observed as high as 12,000 
> imapd processes.  In that particular case, 1 user had 2 imap clients 
> accounting for near 6,000 of those connections.  We've attempted to 
> limit these high numbers using the following imapd.conf values:
> 
> maxlogins_per_host: 50
> maxlogins_per_user: 30
> tcp_keepalive: 1
> tcp_keepalive_cnt: 1
> tcp_keepalive_idle: 30
> tcp_keepalive_intvl: 900
> 
> However, it seems that once these were reached, no new connections were 
> permitted and resulted in all manner of user complaints about not being 
> able to get at their email.
> 
> Any ideas on this "status 71" issue?  Could an upgrade to 2.5.10 
> possibly address this?  Thanks!

https://www.freebsd.org/cgi/man.cgi?query=sysexits

     EX_OSERR (71)         An operating system error has been detected.  This
                           is intended to be used for such things as ``cannot
                           fork'', ``cannot create pipe'', or the like.  It
                           includes things like getuid returning a user that
                           does not exist in the passwd file.

So the question is: what failed?  Is there anything earlier in the log to suggest
what the imapd was doing when it died?

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm