Seg faults in lmtpd in Cyrus 2.3.7 - gdb backtrace
Lenny
rahl at mindcryme.com
Wed Aug 9 13:19:28 EDT 2006
Hello,
I finally have a gdb backtrace of the problem. This happens on the
frontend (We have a murder setup with virtual domains) when it tries
to deliver mail via lmtp. But only sometimes. It could go days and not
have the problem, and when lmtpd DOES start crashing, it'll eventually
start delivering fine again in 5-10 minutes.
The backtrace:
Using host libthread_db library "/lib/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread -1208453440 (LWP 28815)]
0x00fe97a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208453440 (LWP 28815)]
0x0809df30 in cyrus_acl_myrights (auth_state=0x0,
acl=0x1ff5 <Address 0x1ff5 out of bounds>) at acl_afs.c:73
73 acl_afs.c: No such file or directory.
in acl_afs.c
#0 0x0809df30 in cyrus_acl_myrights (auth_state=0x0,
acl=0x1ff5 <Address 0x1ff5 out of bounds>) at acl_afs.c:73
#1 0x0804fe45 in verify_user (user=0x86e0be8 "morich",
domain=0x86e0bef "XXXXX.net", mailbox=0x0, quotacheck=1357, authstate=0x0)
at lmtpd.c:1089
#2 0x080526de in process_recipient (addr=0x86e0ba9 "", namespace=0x8141b60,
ignorequota=0, verify_user=0x804fb42 <verify_user>, msg=0x86d14f8)
at lmtpengine.c:892
#3 0x08053d4d in lmtpmode (func=0x8139600, pin=0x86d8288, pout=0x86d97c8,
fd=0) at lmtpengine.c:1519
#4 0x0804e011 in service_main (argc=1, argv=0x86cf008, envp=0xbfe17828)
at lmtpd.c:287
#5 0x0804dc1a in main (argc=4, argv=0xbfe17814, envp=0xbfe17828)
at service.c:532
In case it's important, I'm using the following imapd.conf:
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyrus cyrus at XXXX.net cyrus at XXXX.net lennyl at XXXX.net cyrus at XXXX.com
sieveuserhomedir: no
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN DIGEST-MD5
tls_key_file: disabled
virtdomains: yes
defaultdomain: XXXX.com
proxy_authname: cyrus
proxy_realm: XXXX.net
cyr-backend1_password: XXXX
cyr-backend2_password: XXXX
cyr-backend3_password: XXXX
cyr-backend4_password: XXXX
cyr-backend5_password: XXXX
mupdate_server: 209.XX.XXX.XX
mupdate_authname: cyrus
mupdate_password: Nasd#q!
allowusermoves: true
dracinterval: 5
drachost: outbound1.XXXXX.net
dracinterval2: 5
drachost2: outbound2.XXXXX.net
drachost3: 0
dracinterval3: 0
lmtp_over_quota_perm_failure: 1
quotawarn: 80
unixhierarchysep: yes
lmtp_downcase_rcpt: 1
duplicate_db: skiplist
ptscache_db: skiplist
tlscache_db: skiplist
allowapop: 0
allowplaintext: 1
client_timeout: 240
expunge_mode: immediate
poptimeout: 40
I'm hoping someone can help me out here:)
Thanks.
Lenny
--
"Wisdom is to a man an infinite Treasure" - Anonymous
Quoting Lenny <rahl at mindcryme.com>:
> Hello,
>
> I'm currently using Simon's rpms on a Centos 4 murder setup. For
> awhile now, I've been getting (occasionally) 15-20 minutes worth of
> "Deferred" messages from sendmail. I never believed it was sendmail,
> but I've been having issues proving that. We managed to get the
> debug_command going in Cyrus, and have now found that it's lmtpd
> that's seg faulting, but I'm still confused as to why.
>
> This happens, like I said, randomly for 15-20 minutes at a time. It
> will eventually start delivering fine on it's own. As a matter of
> fact, any attempt at restarting services (cyrus, sendmail, etc) or
> rebooting the server has NO effect (it will just keep Deferring until
> it starts randomly delivering again).
>
> At any rate, I so far have telemetry and strace logs and will paste an
> example here. I hope to recompile with debugging and have a gdb
> backtrace soon as well, but was wondering if anyone had ideas.
>
> This has been happening for awhile now. It even happened occasionally
> in 2.2.x tho nowhere near as bad as 2.3.x has been. This also seems to
> happen after the "Rcpt to:", either before or during the DATA section.
>
> Example telemetry log:
>
>> 1154532184>220 XXXXXXXXX.net LMTP Cyrus v2.3.7-Invoca-RPM-2.3.7-2.LN1 ready
> <1154532184<LHLO XXXXXXXXXXX.net
>> 1154532184>250-cXXXXXXXXX.net
> 250-8BITMIME
> 250-ENHANCEDSTATUSCODES
> 250-PIPELINING
> 250-SIZE
> 250-AUTH EXTERNAL
> 250 IGNOREQUOTA
> <1154532184<MAIL From:<alerts at XXXXXXXXXX.com> SIZE=2109
>> 1154532184>250 2.1.0 ok
> <1154532184<RCPT To:<stckayak at XXXXXXXXXXX.net>
> RCPT To:<jrforbes at XXXXXXXX.net>
> DATA
>
> (and that's the last thing in the telemetry log).
>
> Excerpt from strace:
>
> 11:23:04.144071 send(5, "<23>Aug 2 11:23:04 lmtpunix[100"..., 73,
> MSG_NOSIGNAL) = 73
> 11:23:04.144318 open("/var/lib/imap/log/postman/10002",
> O_WRONLY|O_APPEND|O_CREAT, 0644) = 8
> 11:23:04.144569 time(NULL) = 1154532184
> 11:23:04.144788 write(8, "---------- postman Wed Aug 2 11"..., 45) = 45
> 11:23:04.145044 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout)
> 11:23:04.145206 time([1154532184]) = 1154532184
> 11:23:04.145351 write(8, ">1154532184>", 12) = 12
> 11:23:04.145508 write(8, "220 XXXXXXXXXXX.net LMTP Cyr"..., 72) = 72
> 11:23:04.145704 write(1, "220 XXXXXXXXXXX.net LMTP Cyr"..., 72) = 72
> 11:23:04.146082 time(NULL) = 1154532184
> 11:23:04.146225 select(1, [0], NULL, NULL, {360, 0}) = 1 (in [0], left
> {360, 0})
> 11:23:04.146408 time(NULL) = 1154532184
> 11:23:04.146532 time(NULL) = 1154532184
> 11:23:04.146684 read(0, "LHLO XXXXXXXXXXX.net\r\n", 4096) = 26
> 11:23:04.146856 time([1154532184]) = 1154532184
> 11:23:04.147030 write(8, "<1154532184<", 12) = 12
> 11:23:04.147187 write(8, "LHLO XXXXXXXXXX.net\r\n", 26) = 26
> 11:23:04.147430 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout)
> 11:23:04.147592 time([1154532184]) = 1154532184
> 11:23:04.147763 write(8, ">1154532184>", 12) = 12
> 11:23:04.147914 write(8, "250-XXXXXXXXXXX.net\r\n250-8BI"..., 126) = 126
> 11:23:04.148065 write(1, "250-XXXXXXXXXXX.net\r\n250-8BI"..., 126) = 126
> 11:23:04.149743 time(NULL) = 1154532184
> 11:23:04.149893 select(1, [0], NULL, NULL, {360, 0}) = 1 (in [0], left
> {360, 0})
> 11:23:04.150075 time(NULL) = 1154532184
> 11:23:04.150198 time(NULL) = 1154532184
> 11:23:04.150317 read(0, "MAIL From:<alerts at XXXXXXXX."..., 4096) = 48
> 11:23:04.150484 time([1154532184]) = 1154532184
> 11:23:04.150658 write(8, "<1154532184<", 12) = 12
> 11:23:04.150822 write(8, "MAIL From:<alerts at XXXXXXX."..., 48) = 48
> 11:23:04.151030 open("/var/lib/imap/msg/shutdown", O_RDONLY) = -1
> ENOENT (No such file or directory)
> 11:23:04.151314 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout)
> 11:23:04.151477 time([1154532184]) = 1154532184
> 11:23:04.151619 write(8, ">1154532184>", 12) = 12
> 11:23:04.151811 write(8, "250 2.1.0 ok\r\n", 14) = 14
> 11:23:04.151961 write(1, "250 2.1.0 ok\r\n", 14) = 14
> 11:23:04.153298 time(NULL) = 1154532184
> 11:23:04.153442 select(1, [0], NULL, NULL, {360, 0}) = 1 (in [0], left
> {360, 0})
> 11:23:04.153619 time(NULL) = 1154532184
> 11:23:04.153779 time(NULL) = 1154532184
> 11:23:04.153902 read(0, "RCPT To:<stckayak at XXXXXXX.net>\r\n"..., 4096) = 70
> 11:23:04.154068 time([1154532184]) = 1154532184
> 11:23:04.154215 write(8, "<1154532184<", 12) = 12
> 11:23:04.154364 write(8, "RCPT To:<stckayak at XXXXXXX.net>\r\n"..., 70) = 70
> 11:23:04.154539 open("/var/lib/imap/msg/shutdown", O_RDONLY) = -1
> ENOENT (No such file or directory)
> 11:23:04.154809 select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout)
> 11:23:04.155000 write(6,
> "\0\0\0:\305\"K\341U\326w:u4\273hx\263\274\363\374\246\24"..., 62) = 62
> 11:23:04.155671 time(NULL) = 1154532184
> 11:23:04.155818 read(6,
> "\0\0\0\246\205\3669\356\212\360\351S\260t\305\233\274@"..., 4096) = 170
> 11:23:04.156045 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
>
> I'll try to post a backtrace of a crashing lmtpd process as soon as I
> can, but does anyone have any ideas as to where I can start looking?
>
> Thanks.
>
> Lenny
> --
> "Wisdom is to a man an infinite Treasure" - Anonymous
>
>
>
>
>
> ----
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
More information about the Info-cyrus
mailing list