cyr_expire suddenly segfaulting

Sun Mar 12 13:24:27 EDT 2017

Hi,
I've been running cyrus-imap for a few years not. I am using 2.4.17-8 on 
ClearOS (a Centos derivative). I've been running with "delprune  
cmd="cyr_expire -E 1 -X 7 -D 7" at=0001" in cyrus.conf, but in the last 
week or so it has started segfaulting. I've tried moving the command to 
cron.daily "/usr/lib/cyrus-imapd/cyr_expire -E 1 -X 7 -D 7" but it still 
segfaulting.

In /var/log/maillog I get:
Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox user.mail-ebay
Mar 12 03:15:53 server master[15064]: setrlimit: Unable to set file 
descriptors limit to -1: Operation not permitted
Mar 12 03:15:53 server master[15064]: retrying with 4096 (current max)
Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox 
user.mail-ebay.Drafts
Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox user.nick
Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox user.ourfamily
Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox 
user.ourfamily.Drafts
Mar 12 03:15:54 server cyr_expire[15065]: Expunged 0 out of 9150 
messages from 57 mailboxes
Mar 12 03:15:54 server cyr_expire[15065]: duplicate_prune: pruning back 
0.00 days
Mar 12 03:15:54 server ctl_cyrusdb[15068]: recovering cyrus databases
Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed 
/var/lib/imap/mailboxes.db (57 records, 4816 bytes) in 0 seconds
Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed 
/var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
Mar 12 03:15:55 server ctl_cyrusdb[15068]: done recovering cyrus databases
Mar 12 03:15:55 server master[15064]: unable to setsocketopt(IP_TOS): 
Operation not supported
Mar 12 03:15:55 server ctl_cyrusdb[15073]: checkpointing cyrus databases
Mar 12 03:15:55 server ctl_cyrusdb[15073]: done checkpointing cyrus 
databases
Mar 12 03:15:55 server imaps[15075]: error sending to idled: 3
Mar 12 03:15:55 server imap[15074]: error sending to idled: 3
Mar 12 03:15:55 server imap[15078]: error sending to idled: 3
Mar 12 03:15:55 server imap[15076]: error sending to idled: 3
Mar 12 03:15:55 server imaps[15081]: error sending to idled: 3
Mar 12 03:15:55 server imap[15080]: error sending to idled: 3
Mar 12 03:15:55 server imaps[15077]: error sending to idled: 3
Mar 12 03:15:55 server imaps[15079]: error sending to idled: 3
Mar 12 03:15:56 server tls_prune[15071]: skiplist: checkpointed 
/var/lib/imap/tls_sessions.db (575 records, 122352 bytes) in 0 seconds
Mar 12 03:15:56 server cyr_expire[15072]: skiplist: checkpointed 
/var/lib/imap/deliver.db (37 records, 5464 bytes) in 0 seconds
Mar 12 03:15:56 server cyr_expire[15072]: Expunged 0 out of 9112 
messages from 57 mailboxes
Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: pruning back 
3.00 days
Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: purged 0 out 
of 37 entries

The "error sending to idled: 3" is probably a red herring as cyrus-imap 
has just restarted following a db backup, so earlier idled connections 
have broken.

Correspondingly in /var/log/messages I get:
Mar 12 03:15:56 server kernel: cyr_expire[15065]: segfault at 
7f9c8a79be78 ip 00007f9c8a91c5ed sp 00007ffd44b51d38 error 4 in 
cyr_expire[7f9c8a8c9000+15c000]

It may also be worth mentioning that on some nights, like the night 
before the segfault, in /var/log/maillog I see:
Mar 11 03:42:09 server cyr_expire[3237]: Expunged 0 out of 9145 messages 
from 61 mailboxes
Mar 11 03:42:09 server cyr_expire[3237]: duplicate_prune: pruning back 
1.00 days
Mar 11 03:42:11 server cyr_expire[3237]: duplicate_prune: purged 58 out 
of 147 entries
Mar 11 03:45:39 server ctl_cyrusdb[25201]: checkpointing cyrus databases
Mar 11 03:45:39 server ctl_cyrusdb[25201]: done checkpointing cyrus 
databases
Mar 11 03:49:57 server master[31633]: exiting on SIGTERM/SIGINT
Mar 11 03:50:58 server master[27624]: setrlimit: Unable to set file 
descriptors limit to -1: Operation not permitted
Mar 11 03:50:58 server master[27624]: retrying with 4096 (current max)
Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox user.mail-ebay
Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox user.ourfamily
Mar 11 03:50:59 server cyr_expire[27625]: Repacking mailbox 
user.ourfamily.Drafts
Mar 11 03:50:59 server ctl_cyrusdb[27628]: recovering cyrus databases
Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed 
/var/lib/imap/mailboxes.db (61 records, 5148 bytes) in 0 seconds
Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed 
/var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
Mar 11 03:50:59 server ctl_cyrusdb[27628]: done recovering cyrus databases
Mar 11 03:50:59 server cyr_expire[27625]: Expunged 0 out of 9145 
messages from 61 mailboxes
Mar 11 03:50:59 server cyr_expire[27625]: duplicate_prune: pruning back 
0.00 days
Mar 11 03:50:59 server master[27624]: unable to setsocketopt(IP_TOS): 
Operation not supported
Mar 11 03:50:59 server ctl_cyrusdb[27633]: checkpointing cyrus databases
Mar 11 03:50:59 server ctl_cyrusdb[27633]: done checkpointing cyrus 
databases
Mar 11 03:51:00 server tls_prune[27631]: skiplist: checkpointed 
/var/lib/imap/tls_sessions.db (303 records, 64348 bytes) in 1 second
Mar 11 03:51:00 server cyr_expire[27632]: skiplist: checkpointed 
/var/lib/imap/deliver.db (84 records, 12424 bytes) in 0 seconds
Mar 11 03:51:00 server cyr_expire[27632]: Expunged 0 out of 9109 
messages from 61 mailboxes
Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: pruning back 
3.00 days
Mar 11 03:51:00 server tls_prune[27631]: tls_prune: purged 3 out of 303 
entries
Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: purged 0 out 
of 83 entries
Mar 11 03:51:04 server cyr_expire[27625]: duplicate_prune: purged 89 out 
of 89 entries
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1581 File 
handles still open at environment close
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file 
handle: /var/lib/imap/db/__db.001
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file 
handle: /var/lib/imap/db/__db.002
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file 
handle: /var/lib/imap/db/__db.003
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Mar 11 03:51:04 server cyr_expire[27625]: DBERROR: critical database 
situation

I'm afraid I've absolutely no idea how to troubleshoot this. Can anyone 
please help?

Thanks,

Nick