cyr_expire suddenly segfaulting
Kenneth Marshall
ktm at rice.edu
Mon Mar 13 08:36:40 EDT 2017
On Mon, Mar 13, 2017 at 11:32:38AM +1100, Bron Gondwana wrote:
> So everything is 2.4 defaults, which are:
>
> brong at wot:~/src/cyrus-imapd$ grep '_db"' lib/imapoptions
> { "annotation_db", "skiplist", STRINGLIST("berkeley", "berkeley-hash", "skiplist")}
> { "duplicate_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync", "berkeley-hash", "berkeley-hash-nosync", "skiplist", "sql")}
> { "mboxkey_db", "skiplist", STRINGLIST("berkeley", "skiplist") }
> { "mboxlist_db", "skiplist", STRINGLIST("flat", "berkeley", "berkeley-hash", "skiplist")}
> { "ptscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-hash", "skiplist")}
> { "quota_db", "quotalegacy", STRINGLIST("flat", "berkeley", "berkeley-hash", "skiplist", "sql", "quotalegacy")}
> { "seenstate_db", "skiplist", STRINGLIST("flat", "berkeley", "berkeley-hash", "skiplist")}
> { "subscription_db", "flat", STRINGLIST("flat", "berkeley", "berkeley-hash", "skiplist")}
> { "statuscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync", "berkeley-hash", "berkeley-hash-nosync", "skiplist") }
> { "tlscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync", "berkeley-hash", "berkeley-hash-nosync", "skiplist", "sql")}
> { "userdeny_db", "flat", STRINGLIST("flat", "berkeley", "berkeley-hash", "skiplist", "sql")}
>
> So you shouldn't have any berkeley databases at all. It should be perfectly safe to shut down Cyrus, delete the $confdir/db folder, and restart Cyrus.
>
> that's:
>
> rm -rf /var/lib/imap/db/
>
> While Cyrus is shut down :)
>
> (Normal warnings about backing up your stuff and making sure that Centos packages aren't doing something different with defaults than upstream)
>
> Cheers,
>
> Bron.
>
> On Mon, 13 Mar 2017, at 08:28, Nick Howitt wrote:
> >
> >
> > On 12/03/2017 21:09, Patrick Boutilier wrote:
> > > On 03/12/2017 02:24 PM, Nick Howitt wrote:
> > >> Hi,
> > >> I've been running cyrus-imap for a few years not. I am using 2.4.17-8 on
> > >> ClearOS (a Centos derivative). I've been running with "delprune
> > >> cmd="cyr_expire -E 1 -X 7 -D 7" at=0001" in cyrus.conf, but in the last
> > >> week or so it has started segfaulting. I've tried moving the command to
> > >> cron.daily "/usr/lib/cyrus-imapd/cyr_expire -E 1 -X 7 -D 7" but it still
> > >> segfaulting.
> > >>
> > >> In /var/log/maillog I get:
> > >> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox
> > >> user.mail-ebay
> > >> Mar 12 03:15:53 server master[15064]: setrlimit: Unable to set file
> > >> descriptors limit to -1: Operation not permitted
> > >> Mar 12 03:15:53 server master[15064]: retrying with 4096 (current max)
> > >> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox
> > >> user.mail-ebay.Drafts
> > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox user.nick
> > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox
> > >> user.ourfamily
> > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox
> > >> user.ourfamily.Drafts
> > >> Mar 12 03:15:54 server cyr_expire[15065]: Expunged 0 out of 9150
> > >> messages from 57 mailboxes
> > >> Mar 12 03:15:54 server cyr_expire[15065]: duplicate_prune: pruning back
> > >> 0.00 days
> > >> Mar 12 03:15:54 server ctl_cyrusdb[15068]: recovering cyrus databases
> > >> Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed
> > >> /var/lib/imap/mailboxes.db (57 records, 4816 bytes) in 0 seconds
> > >> Mar 12 03:15:55 server ctl_cyrusdb[15068]: skiplist: checkpointed
> > >> /var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
> > >> Mar 12 03:15:55 server ctl_cyrusdb[15068]: done recovering cyrus
> > >> databases
> > >> Mar 12 03:15:55 server master[15064]: unable to setsocketopt(IP_TOS):
> > >> Operation not supported
> > >> Mar 12 03:15:55 server ctl_cyrusdb[15073]: checkpointing cyrus databases
> > >> Mar 12 03:15:55 server ctl_cyrusdb[15073]: done checkpointing cyrus
> > >> databases
> > >> Mar 12 03:15:55 server imaps[15075]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imap[15074]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imap[15078]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imap[15076]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imaps[15081]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imap[15080]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imaps[15077]: error sending to idled: 3
> > >> Mar 12 03:15:55 server imaps[15079]: error sending to idled: 3
> > >> Mar 12 03:15:56 server tls_prune[15071]: skiplist: checkpointed
> > >> /var/lib/imap/tls_sessions.db (575 records, 122352 bytes) in 0 seconds
> > >> Mar 12 03:15:56 server cyr_expire[15072]: skiplist: checkpointed
> > >> /var/lib/imap/deliver.db (37 records, 5464 bytes) in 0 seconds
> > >> Mar 12 03:15:56 server cyr_expire[15072]: Expunged 0 out of 9112
> > >> messages from 57 mailboxes
> > >> Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: pruning back
> > >> 3.00 days
> > >> Mar 12 03:15:56 server cyr_expire[15072]: duplicate_prune: purged 0 out
> > >> of 37 entries
> > >>
> > >> The "error sending to idled: 3" is probably a red herring as cyrus-imap
> > >> has just restarted following a db backup, so earlier idled connections
> > >> have broken.
> > >>
> > >> Correspondingly in /var/log/messages I get:
> > >> Mar 12 03:15:56 server kernel: cyr_expire[15065]: segfault at
> > >> 7f9c8a79be78 ip 00007f9c8a91c5ed sp 00007ffd44b51d38 error 4 in
> > >> cyr_expire[7f9c8a8c9000+15c000]
> > >>
> > >> It may also be worth mentioning that on some nights, like the night
> > >> before the segfault, in /var/log/maillog I see:
> > >> Mar 11 03:42:09 server cyr_expire[3237]: Expunged 0 out of 9145 messages
> > >> from 61 mailboxes
> > >> Mar 11 03:42:09 server cyr_expire[3237]: duplicate_prune: pruning back
> > >> 1.00 days
> > >> Mar 11 03:42:11 server cyr_expire[3237]: duplicate_prune: purged 58 out
> > >> of 147 entries
> > >> Mar 11 03:45:39 server ctl_cyrusdb[25201]: checkpointing cyrus databases
> > >> Mar 11 03:45:39 server ctl_cyrusdb[25201]: done checkpointing cyrus
> > >> databases
> > >> Mar 11 03:49:57 server master[31633]: exiting on SIGTERM/SIGINT
> > >> Mar 11 03:50:58 server master[27624]: setrlimit: Unable to set file
> > >> descriptors limit to -1: Operation not permitted
> > >> Mar 11 03:50:58 server master[27624]: retrying with 4096 (current max)
> > >> Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox
> > >> user.mail-ebay
> > >> Mar 11 03:50:58 server cyr_expire[27625]: Repacking mailbox
> > >> user.ourfamily
> > >> Mar 11 03:50:59 server cyr_expire[27625]: Repacking mailbox
> > >> user.ourfamily.Drafts
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27628]: recovering cyrus databases
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed
> > >> /var/lib/imap/mailboxes.db (61 records, 5148 bytes) in 0 seconds
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27628]: skiplist: checkpointed
> > >> /var/lib/imap/annotations.db (0 records, 144 bytes) in 0 seconds
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27628]: done recovering cyrus
> > >> databases
> > >> Mar 11 03:50:59 server cyr_expire[27625]: Expunged 0 out of 9145
> > >> messages from 61 mailboxes
> > >> Mar 11 03:50:59 server cyr_expire[27625]: duplicate_prune: pruning back
> > >> 0.00 days
> > >> Mar 11 03:50:59 server master[27624]: unable to setsocketopt(IP_TOS):
> > >> Operation not supported
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27633]: checkpointing cyrus databases
> > >> Mar 11 03:50:59 server ctl_cyrusdb[27633]: done checkpointing cyrus
> > >> databases
> > >> Mar 11 03:51:00 server tls_prune[27631]: skiplist: checkpointed
> > >> /var/lib/imap/tls_sessions.db (303 records, 64348 bytes) in 1 second
> > >> Mar 11 03:51:00 server cyr_expire[27632]: skiplist: checkpointed
> > >> /var/lib/imap/deliver.db (84 records, 12424 bytes) in 0 seconds
> > >> Mar 11 03:51:00 server cyr_expire[27632]: Expunged 0 out of 9109
> > >> messages from 61 mailboxes
> > >> Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: pruning back
> > >> 3.00 days
> > >> Mar 11 03:51:00 server tls_prune[27631]: tls_prune: purged 3 out of 303
> > >> entries
> > >> Mar 11 03:51:00 server cyr_expire[27632]: duplicate_prune: purged 0 out
> > >> of 83 entries
> > >> Mar 11 03:51:04 server cyr_expire[27625]: duplicate_prune: purged 89 out
> > >> of 89 entries
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1581 File
> > >> handles still open at environment close
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
> > >> handle: /var/lib/imap/db/__db.001
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
> > >> handle: /var/lib/imap/db/__db.002
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file
> > >> handle: /var/lib/imap/db/__db.003
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB0060 PANIC:
> > >> fatal region error detected; run recovery
> > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR: critical database
> > >> situation
> > >>
> > >> I'm afraid I've absolutely no idea how to troubleshoot this. Can anyone
> > >> please help?
> > >>
> > >> Thanks,
> > >>
> > >> Nick
> > >
> > >
> > > Looks like you might have a corrupted BerlkeyDB . What is the output of ?
> > >
> > > grep _db /etc/imapd.conf
> > Nothing? I can't see any alternative file that may hold this info.
> > imapd.conf is:
> >
> > configdirectory: /var/lib/imap
> > partition-default: /var/spool/imap
> > admins: root
> > sievedir: /var/lib/imap/sieve
> > sendmail: /usr/sbin/sendmail
> > hashimapspool: true
> > sasl_pwcheck_method: saslauthd
> > sasl_mech_list: PLAIN
> > tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
> > tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
> > tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
> >
> > flushseenstate: 1
> > allowplaintext: yes
> > reject8bit: no
> > munge8bit: no
> > lmtp_over_quota_perm_failure: 1
> > timeout: 30
> > imapidlepoll: 60
> > idlesocket: /var/lib/imap/socket/idle
> > lmtpsocket: /var/lib/imap/socket/lmtp
> > allowapop: no
> > altnamespace: 0
> > unixhierarchysep: yes
> > lmtp_downcase_rcpt: yes
> > username_tolower: 1
> > autocreatequota: -1
> > createonpost: 1
> > virtdomains: off
> > expire-days: 1
> > expunge-days: 1
> > delete-days: 7
> >
Hi Nick and Bron,
This is the same database that I mentioned in my Email about BerkeleyDB
in my system, when I do not have one defined either. I did have issues
with connections until it was repaired. I was trying to find out why
it was even in use, as well.
Regards,
Ken
More information about the Info-cyrus
mailing list