Help please, cyrus imapd production system down

mikeegg1 mikeegg1 at mac.com
Wed Mar 26 10:29:37 EDT 2008 

Sometime during the night something happend to my imap store about (23:19 CST). I have executed 'ctl_mboxlist -u < $backup' from two days ago to recover the mboxlist and have done a 'reconstruct -r user.$USER' for a few users to get their $USER.seen folders working again. The process lmtpunix is aborting, I cannot receive email (sendmail to deliver). I can still send mail out (through sendmail) so that part's working.

I'm concerned that I need to recover my primary imap databases.

Can someone help me diagnose what's going on and how to fix it (and what happened)?

I am a small site with less than 30 email accounts in cyrus.

There were no cyrus, sendmail, nor other email changes yesterday.

Mike

Fedora Core 5 (current patches)
$ rpm -qa | grep cyrus
cyrus-imapd-2.3.1-2.8.fc5
cyrus-imapd-devel-2.3.1-2.8.fc5
cyrus-imapd-utils-2.3.1-2.8.fc5
cyrus-sasl-2.1.21-10
cyrus-sasl-devel-2.1.21-10
cyrus-sasl-ldap-2.1.21-10
cyrus-sasl-lib-2.1.21-10

$ rpm -qa | grep db
db4-4.3.29-8.fc5
db4-devel-4.3.29-8.fc5
db4-utils-4.3.29-8.fc5
dbus-0.62-0.2.fc5
dbus-devel-0.62-0.2.fc5
dbus-glib-0.62-0.2.fc5
dbus-python-0.62-0.2.fc5
dbus-sharp-0.62-0.2.fc5
dbus-x11-0.62-0.2.fc5
dhcdbd-1.15-2.FC5
gdb-6.3.0.0-1.134.fc5
gdbm-1.8.0-26.2
gdbm-devel-1.8.0-26.2
jakarta-commons-dbcp-1.2.1-3jpp_3fc
libdbi-0.8.1-1.2.1
libdbi-dbd-mysql-0.8.1a-1.2.1
libdbi-drivers-0.8.1a-1.2.1
mysql-connector-odbc-3.51.12-1.2.1
nss_db-2.2-35

entries from /var/log/maillog
Mar 26 08:59:23 elo master[20695]: service lmtpunix pid 23909 in READY state: terminated abnormally
Mar 26 08:59:23 elo master[23979]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:23 elo lmtpunix[23979]: executed
Mar 26 08:59:24 elo sendmail[22334]: m2QDfxl0022334: ruleset=check_mail, arg1=<hlmbpzwby at countryfordonline.com>, relay=localhost [127.0.
0.1], reject=451 4.1.8 Domain of sender address hlmbpzwby at countryfordonline.com does not resolve
Mar 26 08:59:24 elo sm-msp-queue[21753]: m2LNUjr6004989: to=$USER@$DOMAIN, delay=4+14:28:39, xdelay=00:00:02, mailer=relay, pri=97
52409, relay=[127.0.0.1], dsn=4.0.0, stat=Deferred: 451 4.1.8 Domain of sender address hlmbpzwby at countryfordonline.com does not resolve
Mar 26 08:59:25 elo pop3[23212]: accepted connection
Mar 26 08:59:25 elo master[23980]: about to exec /usr/lib/cyrus-imapd/pop3d
Mar 26 08:59:25 elo pop3[23980]: executed
Mar 26 08:59:25 elo pop3[23212]: login: $MACHINE.$DOMAIN [10.1.2.77] $USER plaintext User logged in
Mar 26 08:59:29 elo imap[20839]: SQUAT failed to open index file
Mar 26 08:59:29 elo imap[20839]: SQUAT failed
Mar 26 08:59:29 elo sendmail[23958]: m2QDxBv3023958: from=<IOR at spvyltah.info>, size=6096, class=0, nrcpts=1, msgid=<E1JeW9V-00018C-OB at ce
valvorto41.vauntstrand.com>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:30 elo sendmail[23984]: m2QDxBv3023958: to=<$USER@$DOMAIN>, delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=126
096, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:32 elo master[20695]: process 23955 exited, signaled to death by 7
Mar 26 08:59:32 elo master[20695]: service lmtpunix pid 23955 in READY state: terminated abnormally
Mar 26 08:59:32 elo master[23988]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:32 elo lmtpunix[23988]: executed
Mar 26 08:59:32 elo sendmail[23957]: m2QDxACR023957: from=<uliedtke at logokett.com>, size=8641, class=0, nrcpts=1, msgid=<151501c88f49$94a
d7780$7c475ec8 at Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:32 elo sendmail[23959]: m2QDxDQY023959: from=<NoelleselectmanGoldsmith at blazersedge.com>, size=6314, class=0, nrcpts=1, msgi
d=11fa701c88f49$97279f40$57e39cca at SN2564177124, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:33 elo sendmail[23990]: m2QDxACR023957: to=<fmpcaqfc@$DOMAIN>, delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2, pri=1
28641, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:33 elo sendmail[23992]: m2QDxDQY023959: to=<$USER@$DOMAIN>, delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=1263
14, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:33 elo sendmail[23938]: m2QDwxuZ023938: from=<n at oae.onkyo.com>, size=11014, class=0, nrcpts=1, msgid=<000601c88f49$04b7aef1
$413589ac at hmwrugyt>, proto=ESMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:34 elo sendmail[23994]: m2QDwxuZ023938: to=<$USER@$DOMAIN>, delay=00:00:27, xdelay=00:00:00, mailer=cyrusv2, pri=13
1014, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:40 elo sendmail[23967]: m2QDxJCu023967: from=<uliedtke at logokett.com>, size=7569, class=0, nrcpts=1, msgid=<154301c88f49$98c
41130$7c475ec8 at Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:40 elo master[20695]: process 23916 exited, signaled to death by 7
Mar 26 08:59:40 elo master[20695]: service lmtpunix pid 23916 in READY state: terminated abnormally
Mar 26 08:59:40 elo master[23998]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:40 elo lmtpunix[23998]: executed
Mar 26 08:59:40 elo sendmail[23997]: m2QDxJCu023967: to=$USER, delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2, pri=37785, relay=local



$ db_verify deliver.db  2>&1 | head
db_verify: deliver.db: unexpected file type or format
db_verify: Page 0: pgno incorrectly set to 1953720684
db_verify: Page 0: bad magic number 543582572
db_verify: Page 0: bad page size 16777216
db_verify: Page 1: bad page number 1206496269
db_verify: Page 1: bad page type 255
db_verify: Page 1: unknown page type 255
db_verify: Page 2: bad page number 1701145449
db_verify: Page 2: bad page type 142
db_verify: Page 2: unknown page type 142
......
db_verify: Page 228: bad page type 109
db_verify: Page 228: unknown page type 109
db_verify: Page 229: bad page number 1465276490
db_verify: Page 229: bad page type 105
db_verify: Page 229: unknown page type 105
db_verify: deliver.db: DB_VERIFY_BAD: Database verification failed


$ file *.db
annotations.db:  Cyrus skiplist DB
deliver.db:      Cyrus skiplist DB
mailboxes.db:    Cyrus skiplist DB
tls_sessions.db: Cyrus skiplist DB


$ file *.db
annotations.db:  Cyrus skiplist DB
deliver.db:      Cyrus skiplist DB
mailboxes.db:    Cyrus skiplist DB
tls_sessions.db: Cyrus skiplist DB
[cyrus at elo ~]$ cat /etc/cyrus.conf
# standard standalone server implementation

START {
  # do not delete this entry!
  recover       cmd="ctl_cyrusdb -r"

  # this is only necessary if using idled for IMAP IDLE
  idled         cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
  # add or remove based on preferences
  imap          cmd="imapd" listen="imap" prefork=10
  imaps         cmd="imapd -s" listen="imaps" prefork=5
  pop3          cmd="pop3d" listen="pop3" prefork=2
  pop3s         cmd="pop3d -s" listen="pop3s" prefork=2
  sieve         cmd="timsieved" listen="sieve" prefork=10

  # these are only necessary if receiving/exporting usenet via NNTP
#  nntp         cmd="nntpd" listen="nntp" prefork=3
#  nntps                cmd="nntpd -s" listen="nntps" prefork=1

  # at least one LMTP is required for delivery
#  lmtp         cmd="lmtpd" listen="localhost:lmtp" prefork=1
  lmtpunix      cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=10

  # this is only necessary if using notifications
#  notify       cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" prefork=1
}

EVENTS {
  # this is required
  checkpoint    cmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune      cmd="cyr_expire -E 3 -X 1" at=0200

  # this is only necessary if caching TLS sessions
  tlsprune      cmd="tls_prune" at=0400

  # mikee 080109 - make sure messages are trimmed
  msgprune1     cmd="ipurge -f -d 2 -X user.%.backup" at=2200
  msgprune2     cmd="ipurge -f -d 5 -X user.%.spam" at=2230
  msgprune3     cmd="ipurge -f -d 4 -X user.%.Trash" at=2300
}


$ cat /etc/imapd.conf
admins: cyrus
allowanonymouslogin: no
allowplaintext: yes
autocreatequota: -1
autocreateinboxfolders: Sent | Drafts | | spam | backup
configdirectory: /var/lib/imap
drachost: localhost
dracinterval: 0
hashimapspool: true
partition-default: /var/spool/imap
poptimeout: 10
postmaster: postmaster
quotawarn: 90
reject8bit: no
sasl_mech_list: PLAIN
sasl_pwcheck_method: saslauthd
sendmail: /usr/lib/sendmail
servername: servername
sieveusehomedir: 0
sievedir: /var/lib/imap/sieve
sieve_maxscripts: 5
sieve_maxscriptsize: 96
timeout: 1440
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
#unixhierarchysep: yes

More information about the Info-cyrus mailing list