cyrus master fails with status 71

[Eric Cunningham]

=============================================================
   Eric Cunningham
   Information Services - http://whoi-it.whoi.edu
   Woods Hole Oceanographic Institution - http://www.whoi.edu
   Woods Hole, MA  02543-1541     phone: (508) 289-2224
   fax: (508) 457-2174           e-mail: ecunningham at whoi.edu
=============================================================

On 10/24/2016 03:45 PM, Bron Gondwana via Info-cyrus wrote:
> On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:
>> Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2
>> (release-p7) ports tree.
>>
>> The cyrus master process is failing periodically (every 1-2 weeks) as
>> follows:
>>
>> Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps
>> path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71
>> Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in
>> READY state: terminated abnormally
>> Oct 22 07:38:48 imap1 master[7767]: too many failures for service
>> imaps/ipv4, disabling until next SIGHUP
>>
>> This prevents new connections by clients until cyrus is restarted.  I've
>> looked around the web but have not seen this issue reported.
>>
>> A little background:
>>
>> Our initial thought on this was that we were running out of listen
>> queues so have upped that incrementally from the default of 32 to a
>> current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l
>> option, with increased kern.ipc.soacceptqueue set to 32768, but that
>> hasn't helped.  Sometimes the "status 71" occurs during periods of light
>> use during off hours, like on Saturday mornings.
>>
>> We have ~1400 imap accounts, though the number of impad processes hovers
>> around 3,000-4,000.  There have been spikes observed as high as 12,000
>> imapd processes.  In that particular case, 1 user had 2 imap clients
>> accounting for near 6,000 of those connections.  We've attempted to
>> limit these high numbers using the following imapd.conf values:
>>
>> maxlogins_per_host: 50
>> maxlogins_per_user: 30
>> tcp_keepalive: 1
>> tcp_keepalive_cnt: 1
>> tcp_keepalive_idle: 30
>> tcp_keepalive_intvl: 900
>>
>> However, it seems that once these were reached, no new connections were
>> permitted and resulted in all manner of user complaints about not being
>> able to get at their email.
>>
>> Any ideas on this "status 71" issue?  Could an upgrade to 2.5.10
>> possibly address this?  Thanks!
>
> https://www.freebsd.org/cgi/man.cgi?query=sysexits
>
>      EX_OSERR (71)         An operating system error has been detected.  This
>                            is intended to be used for such things as ``cannot
>                            fork'', ``cannot create pipe'', or the like.  It
>                            includes things like getuid returning a user that
>                            does not exist in the passwd file.
>
> So the question is: what failed?  Is there anything earlier in the log to suggest
> what the imapd was doing when it died?
>
> Bron.
>

Using the example I posted, I traced back imaps process id 32760 and 
found only this:

Oct 22 07:38:48 imap1 imaps[32760]: accept failed: Software caused 
connection abort

-Eric