Help please, cyrus imapd production system down
Simon Matter
simon.matter at invoca.ch
Wed Mar 26 11:18:10 EDT 2008
> Sometime during the night something happend to my imap store about (23:19
> CST). I have executed 'ctl_mboxlist -u < $backup' from two days ago to
> recover the mboxlist and have done a 'reconstruct -r user.$USER' for a few
> users to get their $USER.seen folders working again. The process lmtpunix
> is aborting, I cannot receive email (sendmail to deliver). I can still
> send mail out (through sendmail) so that part's working.
>
> I'm concerned that I need to recover my primary imap databases.
>
> Can someone help me diagnose what's going on and how to fix it (and what
> happened)?
>
> I am a small site with less than 30 email accounts in cyrus.
>
> There were no cyrus, sendmail, nor other email changes yesterday.
Hm, the 2.3.1 package you use may not be the most stable. Anyway, I guess
you have checked the basic things like full filesystems and so. Then, what
I suggest is
stop cyrus-imapd
mv /var/lib/imap/deliver.db /var/lib/imap/deliver.db.old
mv /var/lib/imap/db /var/lib/imap/db.old
mv /var/lib/imap/db.backup1 /var/lib/imap/db.backup1.old
mv /var/lib/imap/db.backup2 /var/lib/imap/db.backup2.old
do a full offline reconstruct and start cyrus again
Simon
>
> Mike
>
> Fedora Core 5 (current patches)
> $ rpm -qa | grep cyrus
> cyrus-imapd-2.3.1-2.8.fc5
> cyrus-imapd-devel-2.3.1-2.8.fc5
> cyrus-imapd-utils-2.3.1-2.8.fc5
> cyrus-sasl-2.1.21-10
> cyrus-sasl-devel-2.1.21-10
> cyrus-sasl-ldap-2.1.21-10
> cyrus-sasl-lib-2.1.21-10
>
> $ rpm -qa | grep db
> db4-4.3.29-8.fc5
> db4-devel-4.3.29-8.fc5
> db4-utils-4.3.29-8.fc5
> dbus-0.62-0.2.fc5
> dbus-devel-0.62-0.2.fc5
> dbus-glib-0.62-0.2.fc5
> dbus-python-0.62-0.2.fc5
> dbus-sharp-0.62-0.2.fc5
> dbus-x11-0.62-0.2.fc5
> dhcdbd-1.15-2.FC5
> gdb-6.3.0.0-1.134.fc5
> gdbm-1.8.0-26.2
> gdbm-devel-1.8.0-26.2
> jakarta-commons-dbcp-1.2.1-3jpp_3fc
> libdbi-0.8.1-1.2.1
> libdbi-dbd-mysql-0.8.1a-1.2.1
> libdbi-drivers-0.8.1a-1.2.1
> mysql-connector-odbc-3.51.12-1.2.1
> nss_db-2.2-35
>
> entries from /var/log/maillog
> Mar 26 08:59:23 elo master[20695]: service lmtpunix pid 23909 in READY
> state: terminated abnormally
> Mar 26 08:59:23 elo master[23979]: about to exec
> /usr/lib/cyrus-imapd/lmtpd
> Mar 26 08:59:23 elo lmtpunix[23979]: executed
> Mar 26 08:59:24 elo sendmail[22334]: m2QDfxl0022334: ruleset=check_mail,
> arg1=<hlmbpzwby at countryfordonline.com>, relay=localhost [127.0.
> 0.1], reject=451 4.1.8 Domain of sender address
> hlmbpzwby at countryfordonline.com does not resolve
> Mar 26 08:59:24 elo sm-msp-queue[21753]: m2LNUjr6004989: to=$USER@$DOMAIN,
> delay=4+14:28:39, xdelay=00:00:02, mailer=relay, pri=97
> 52409, relay=[127.0.0.1], dsn=4.0.0, stat=Deferred: 451 4.1.8 Domain of
> sender address hlmbpzwby at countryfordonline.com does not resolve
> Mar 26 08:59:25 elo pop3[23212]: accepted connection
> Mar 26 08:59:25 elo master[23980]: about to exec
> /usr/lib/cyrus-imapd/pop3d
> Mar 26 08:59:25 elo pop3[23980]: executed
> Mar 26 08:59:25 elo pop3[23212]: login: $MACHINE.$DOMAIN [10.1.2.77] $USER
> plaintext User logged in
> Mar 26 08:59:29 elo imap[20839]: SQUAT failed to open index file
> Mar 26 08:59:29 elo imap[20839]: SQUAT failed
> Mar 26 08:59:29 elo sendmail[23958]: m2QDxBv3023958:
> from=<IOR at spvyltah.info>, size=6096, class=0, nrcpts=1,
> msgid=<E1JeW9V-00018C-OB at ce
> valvorto41.vauntstrand.com>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN
> [10.1.2.1]
> Mar 26 08:59:30 elo sendmail[23984]: m2QDxBv3023958: to=<$USER@$DOMAIN>,
> delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=126
> 096, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with
> localhost
> Mar 26 08:59:32 elo master[20695]: process 23955 exited, signaled to death
> by 7
> Mar 26 08:59:32 elo master[20695]: service lmtpunix pid 23955 in READY
> state: terminated abnormally
> Mar 26 08:59:32 elo master[23988]: about to exec
> /usr/lib/cyrus-imapd/lmtpd
> Mar 26 08:59:32 elo lmtpunix[23988]: executed
> Mar 26 08:59:32 elo sendmail[23957]: m2QDxACR023957:
> from=<uliedtke at logokett.com>, size=8641, class=0, nrcpts=1,
> msgid=<151501c88f49$94a
> d7780$7c475ec8 at Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN
> [10.1.2.1]
> Mar 26 08:59:32 elo sendmail[23959]: m2QDxDQY023959:
> from=<NoelleselectmanGoldsmith at blazersedge.com>, size=6314, class=0,
> nrcpts=1, msgi
> d=11fa701c88f49$97279f40$57e39cca at SN2564177124, proto=SMTP, daemon=MTA,
> relay=$DMZ.$DOMAIN [10.1.2.1]
> Mar 26 08:59:33 elo sendmail[23990]: m2QDxACR023957:
> to=<fmpcaqfc@$DOMAIN>, delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2,
> pri=1
> 28641, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out
> with localhost
> Mar 26 08:59:33 elo sendmail[23992]: m2QDxDQY023959: to=<$USER@$DOMAIN>,
> delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=1263
> 14, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with
> localhost
> Mar 26 08:59:33 elo sendmail[23938]: m2QDwxuZ023938:
> from=<n at oae.onkyo.com>, size=11014, class=0, nrcpts=1,
> msgid=<000601c88f49$04b7aef1
> $413589ac at hmwrugyt>, proto=ESMTP, daemon=MTA, relay=$DMZ.$DOMAIN
> [10.1.2.1]
> Mar 26 08:59:34 elo sendmail[23994]: m2QDwxuZ023938: to=<$USER@$DOMAIN>,
> delay=00:00:27, xdelay=00:00:00, mailer=cyrusv2, pri=13
> 1014, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with
> localhost
> Mar 26 08:59:40 elo sendmail[23967]: m2QDxJCu023967:
> from=<uliedtke at logokett.com>, size=7569, class=0, nrcpts=1,
> msgid=<154301c88f49$98c
> 41130$7c475ec8 at Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN
> [10.1.2.1]
> Mar 26 08:59:40 elo master[20695]: process 23916 exited, signaled to death
> by 7
> Mar 26 08:59:40 elo master[20695]: service lmtpunix pid 23916 in READY
> state: terminated abnormally
> Mar 26 08:59:40 elo master[23998]: about to exec
> /usr/lib/cyrus-imapd/lmtpd
> Mar 26 08:59:40 elo lmtpunix[23998]: executed
> Mar 26 08:59:40 elo sendmail[23997]: m2QDxJCu023967: to=$USER,
> delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2, pri=37785, relay=local
>
>
>
> $ db_verify deliver.db 2>&1 | head
> db_verify: deliver.db: unexpected file type or format
> db_verify: Page 0: pgno incorrectly set to 1953720684
> db_verify: Page 0: bad magic number 543582572
> db_verify: Page 0: bad page size 16777216
> db_verify: Page 1: bad page number 1206496269
> db_verify: Page 1: bad page type 255
> db_verify: Page 1: unknown page type 255
> db_verify: Page 2: bad page number 1701145449
> db_verify: Page 2: bad page type 142
> db_verify: Page 2: unknown page type 142
> ......
> db_verify: Page 228: bad page type 109
> db_verify: Page 228: unknown page type 109
> db_verify: Page 229: bad page number 1465276490
> db_verify: Page 229: bad page type 105
> db_verify: Page 229: unknown page type 105
> db_verify: deliver.db: DB_VERIFY_BAD: Database verification failed
>
>
> $ file *.db
> annotations.db: Cyrus skiplist DB
> deliver.db: Cyrus skiplist DB
> mailboxes.db: Cyrus skiplist DB
> tls_sessions.db: Cyrus skiplist DB
>
>
> $ file *.db
> annotations.db: Cyrus skiplist DB
> deliver.db: Cyrus skiplist DB
> mailboxes.db: Cyrus skiplist DB
> tls_sessions.db: Cyrus skiplist DB
> [cyrus at elo ~]$ cat /etc/cyrus.conf
> # standard standalone server implementation
>
> START {
> # do not delete this entry!
> recover cmd="ctl_cyrusdb -r"
>
> # this is only necessary if using idled for IMAP IDLE
> idled cmd="idled"
> }
>
> # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
> SERVICES {
> # add or remove based on preferences
> imap cmd="imapd" listen="imap" prefork=10
> imaps cmd="imapd -s" listen="imaps" prefork=5
> pop3 cmd="pop3d" listen="pop3" prefork=2
> pop3s cmd="pop3d -s" listen="pop3s" prefork=2
> sieve cmd="timsieved" listen="sieve" prefork=10
>
> # these are only necessary if receiving/exporting usenet via NNTP
> # nntp cmd="nntpd" listen="nntp" prefork=3
> # nntps cmd="nntpd -s" listen="nntps" prefork=1
>
> # at least one LMTP is required for delivery
> # lmtp cmd="lmtpd" listen="localhost:lmtp" prefork=1
> lmtpunix cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=10
>
> # this is only necessary if using notifications
> # notify cmd="notifyd" listen="/var/lib/imap/socket/notify"
> proto="udp" prefork=1
> }
>
> EVENTS {
> # this is required
> checkpoint cmd="ctl_cyrusdb -c" period=30
>
> # this is only necessary if using duplicate delivery suppression,
> # Sieve or NNTP
> delprune cmd="cyr_expire -E 3 -X 1" at=0200
>
> # this is only necessary if caching TLS sessions
> tlsprune cmd="tls_prune" at=0400
>
> # mikee 080109 - make sure messages are trimmed
> msgprune1 cmd="ipurge -f -d 2 -X user.%.backup" at=2200
> msgprune2 cmd="ipurge -f -d 5 -X user.%.spam" at=2230
> msgprune3 cmd="ipurge -f -d 4 -X user.%.Trash" at=2300
> }
>
>
> $ cat /etc/imapd.conf
> admins: cyrus
> allowanonymouslogin: no
> allowplaintext: yes
> autocreatequota: -1
> autocreateinboxfolders: Sent | Drafts | | spam | backup
> configdirectory: /var/lib/imap
> drachost: localhost
> dracinterval: 0
> hashimapspool: true
> partition-default: /var/spool/imap
> poptimeout: 10
> postmaster: postmaster
> quotawarn: 90
> reject8bit: no
> sasl_mech_list: PLAIN
> sasl_pwcheck_method: saslauthd
> sendmail: /usr/lib/sendmail
> servername: servername
> sieveusehomedir: 0
> sievedir: /var/lib/imap/sieve
> sieve_maxscripts: 5
> sieve_maxscriptsize: 96
> timeout: 1440
> tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
> tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
> tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
> #unixhierarchysep: yes
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
More information about the Info-cyrus
mailing list