[SOLVED] Re: master process exited, signaled to death by 11 service pid in BUSY state: terminated abnormally

[Nestor A. Diaz]

hello again, the problem was with the ldap database, a person modify the 
ldap database, and delete a user entry (memberUid: xxxxx) from a 
'posixGroup', while the user data remains, and  'gidNumber' of the user 
reference the 'posixGroup' above, because of that cyrus always return 
with a segmentation fault.

since i was unable to find a solution and have tried many ways, (ending 
on irratioinals), and found that the problem was on ldap (saldb works), 
i decide to restore an old ldap backup, and i notice that it works, then 
doing a diff againt the latest ldap ldif,  i found the problem !!!  by 
curiosity i decide to test again with the mal formed ldif, and try to 
make a `ls -l /home` and the program returns a segmentation fault, so 
the problem was not cyrus only, it was a mal formed ldap database that 
cause the problem.

Nestor A. Diaz wrote:
> Hello people, starting from friday, cyrus decide not to work again, i 
> have not been able to found the cause for this, i upgrade to cyrus 
> 2.2, hoping it will be fix the problem, but the problem still remain, 
> i use debian gnu/linux sarge and right know i have the cyrus 2.2 
> backport running:
>
> ii  cyrus-admin-2. 2.2.13-8~bpo1  Cyrus mail system (administration 
> tools)
> ii  cyrus-common-2 2.2.13-8~bpo1  Cyrus mail system (common files)
> ii  cyrus-imapd-2. 2.2.13-8~bpo1  Cyrus mail system (IMAP support)
> ii  cyrus21-client 2.1.18-1+sarge Cyrus mail system (test clients)
> ii  cyrus21-doc    2.1.18-1+sarge Cyrus mail system (documentation files)
> ii  libcyrus-imap- 2.2.13-8~bpo1  Interface to Cyrus imap client 
> imclient libr
> ii  libsasl2       2.1.19.dfsg1-0 Authentication abstraction library
> ii  libsasl2-modul 2.1.19.dfsg1-0 Pluggable Authentication Modules for 
> SASL
> ii  sasl2-bin      2.1.19.dfsg1-0 Programs for manipulating the SASL 
> users dat
>
> i authenticate agains ldap, but authentication is ok, because i can 
> supply a valid and a wrong password and cyrus notice that.
>
> when i start cyrus it shows:
>
> Dec 16 15:01:34 pascacio master[19369]: process started
> Dec 16 15:01:34 pascacio master[19370]: about to exec 
> /usr/sbin/ctl_cyrusdb
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19370]: recovering cyrus 
> databases
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19370]: skiplist: recovered 
> /var/lib/cyrus/mailboxes.db (1286 records, 114188 bytes) in 0 seconds
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19370]: skiplist: recovered 
> /var/lib/cyrus/annotations.db (0 records, 144 bytes) in 0 seconds
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19370]: done recovering 
> cyrus databases
> Dec 16 15:01:34 pascacio master[19371]: about to exec 
> /usr/sbin/cyr_expire
> Dec 16 15:01:34 pascacio cyrus/cyr_expire[19371]: duplicate_prune: 
> pruning back 3 days
> Dec 16 15:01:34 pascacio cyrus/cyr_expire[19371]: duplicate_prune: 
> purged 0 out of 727 entries
> Dec 16 15:01:34 pascacio cyrus/cyr_expire[19371]: expunged 0 out of 0 
> messages from 0 mailboxes
> Dec 16 15:01:34 pascacio master[19372]: about to exec /usr/sbin/tls_prune
> Dec 16 15:01:34 pascacio cyrus/tls_prune[19372]: tls_prune: purged 0 
> out of 0 entries
> Dec 16 15:01:34 pascacio master[19369]: WARNING: cannot find 
> executable for service 'pop3' -- ignored
> Dec 16 15:01:34 pascacio master[19369]: ready for work
> Dec 16 15:01:34 pascacio master[19374]: about to exec 
> /usr/sbin/ctl_cyrusdb
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: checkpointing cyrus 
> databases
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: archiving database 
> file: /var/lib/cyrus/annotations.db
> Dec 16 15:01:34 pascacio master[19375]: about to exec 
> /usr/lib/cyrus/bin/notifyd
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: archiving log file: 
> /var/lib/cyrus/db/log.0000000012
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: archiving log file: 
> /var/lib/cyrus/db/log.0000000012
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: archiving database 
> file: /var/lib/cyrus/mailboxes.db
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: archiving log file: 
> /var/lib/cyrus/db/log.0000000012
> Dec 16 15:01:34 pascacio cyrus/notify[19375]: executed
> Dec 16 15:01:34 pascacio cyrus/ctl_cyrusdb[19374]: done checkpointing 
> cyrus databases
> Dec 16 15:01:34 pascacio master[19369]: process 19374 exited, status 0
>
>
> when i try to access a mailbox:
>
> Dec 16 15:02:07 pascacio master[19389]: about to exec 
> /usr/lib/cyrus/bin/imapd
> Dec 16 15:02:07 pascacio cyrus/imaplocalhost[19389]: running external 
> debugger: /usr/bin/gdb -batch -cd=/tmp -x 
> /usr/lib/cyrus/get-backtrace.gdb /usr/
> lib/cyrus/bin/imapd 19389 >/tmp/gdb-backtrace.cyrus.imapd.19389 <&- 
> 2>&1 &
> Dec 16 15:02:07 pascacio cyrus/imaplocalhost[19389]: debugger returned 
> exit status: 0
> Dec 16 15:02:07 pascacio cyrus/imaplocalhost[19389]: executed
> Dec 16 15:02:07 pascacio cyrus/imaplocalhost[19389]: accepted connection
> Dec 16 15:02:07 pascacio master[19369]: process 19389 exited, signaled 
> to death by 11
> Dec 16 15:02:07 pascacio master[19369]: service imaplocalhost pid 
> 19389 in BUSY state: terminated abnormally
>
> This are the contests of /tmp/gdb-backtrace.cyrus.imapd.19389:
>
> (no debugging symbols found)
> Using host libthread_db library "/lib/libthread_db.so.1".
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> 0x403b5788 in waitpid () from /lib/libc.so.6
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x403802d4 in mallopt () from /lib/libc.so.6
> #0  0x403802d4 in mallopt () from /lib/libc.so.6
> #1  0x4037f0df in free () from /lib/libc.so.6
> #2  0x417be63e in ?? ()
> #3  0x08155304 in ?? ()
> #4  0x08155307 in ?? ()
> #5  0x00000000 in ?? ()
> #6  0xbfffcce0 in ?? ()
> #7  0x0814fa70 in ?? ()
> #8  0x417c32d0 in ?? ()
> #9  0xbfffcce8 in ?? ()
> #10 0x417be6a9 in ?? ()
> #11 0x08155304 in ?? ()
> #12 0x41833f78 in ?? ()
> #13 0xbfffccf8 in ?? ()
> #14 0x417b6734 in ?? ()
> #15 0x08155307 in ?? ()
> #16 0x00000003 in ?? ()
> #17 0xbfffccf8 in ?? ()
> #18 0x4179eadf in ?? ()
> #19 0x08156408 in ?? ()
> #20 0x41776888 in ?? ()
> #21 0xbfffcd88 in ?? ()
> #22 0x417718ad in ?? ()
> #23 0x08156408 in ?? ()
> #24 0x08155307 in ?? ()
> #25 0x0000000c in ?? ()
> #26 0x41774955 in ?? ()
> #27 0x41776b00 in ?? ()
> #28 0x00000000 in ?? ()
> #29 0x41774955 in ?? ()
> #30 0x08156408 in ?? ()
> #31 0xbfffcd30 in ?? ()
> #32 0x0000000c in ?? ()
> #33 0x08154fde in ?? ()
> #34 0x0814fa90 in ?? ()
> #35 0x003d6e63 in ?? ()
> #36 0x00000000 in ?? ()
> #37 0x0814f960 in ?? ()
> #38 0x0814f999 in ?? ()
> #39 0x0814f9c7 in ?? ()
> #40 0x00000000 in ?? ()
> #41 0x00000000 in ?? ()
> #42 0x41833f9b in ?? ()
> #43 0x4043d8c0 in ?? () from /lib/libc.so.6
> #44 0xbfffcd5c in ?? ()
> #45 0x41833f78 in ?? ()
> #46 0x41776888 in ?? ()
> #47 0x00000000 in ?? ()
> #48 0x41774c3c in ?? ()
> #49 0xbfffcd88 in ?? ()
> #50 0x4176a7a6 in ?? ()
> #51 0x081520f0 in ?? ()
> #52 0x08152438 in ?? ()
> #53 0x00000005 in ?? ()
> #54 0x41776888 in ?? ()
> #55 0x081545b0 in ?? ()
> #56 0x41774c3c in ?? ()
> #57 0xbfffcdb8 in ?? ()
> #58 0x41771721 in ?? ()
> #59 0x081545b0 in ?? ()
> #60 0x41774955 in ?? ()
> #61 0x4043eb40 in timezone () from /lib/libc.so.6
> #62 0xbfffcf50 in ?? ()
> #63 0x00002712 in ?? ()
> #64 0x00000000 in ?? ()
> #65 0x41776888 in ?? ()
> #66 0x00000000 in ?? ()
> #67 0x41774c3c in ?? ()
> #68 0xbfffcf38 in ?? ()
> #69 0x4176d098 in ?? ()
> #70 0x08152438 in ?? ()
> #71 0x41774955 in ?? ()
> #72 0x4043eb40 in timezone () from /lib/libc.so.6
> #73 0xbfffcf50 in ?? ()
> #74 0x00000000 in ?? ()
> #75 0xbfffce1c in ?? ()
> #76 0x417b6240 in ?? ()
> #77 0x08153438 in ?? ()
> #78 0x00000000 in ?? ()
> #79 0x00000000 in ?? ()
> #80 0x00000000 in ?? ()
> #81 0xfffffffe in ?? ()
> #82 0x08154418 in ?? ()
> #83 0x00000000 in ?? ()
> #84 0xbfffcf4c in ?? ()
> #85 0xbfffcf50 in ?? ()
> #86 0x08154460 in ?? ()
> #87 0x00000000 in ?? ()
> #88 0x00000000 in ?? ()
> #89 0x00000000 in ?? ()
> #90 0x00000000 in ?? ()
> #91 0x00000000 in ?? ()
> #92 0x08154fd8 in ?? ()
> #93 0x41839d00 in ?? ()
> #94 0xbfffcf9c in ?? ()
> #95 0xbfffce40 in ?? ()
> #96 0xbfffcedc in ?? ()
> #97 0x41833828 in ?? ()
> #98 0x0000000d in ?? ()
> #99 0xbfffce40 in ?? ()
> #100 0x41777b00 in ?? ()
> #101 0x00000001 in ?? ()
> #102 0x00000000 in ?? ()
> #103 0x00000000 in ?? ()
> #104 0x00000000 in ?? ()
> #105 0x00000000 in ?? ()
> #106 0x00000000 in ?? ()
> #107 0x00000000 in ?? ()
> #108 0x00000000 in ?? ()
> #109 0x417b6734 in ?? ()
> #110 0x081522b0 in ?? ()
> #111 0x081523b0 in ?? ()
> #112 0x417b6734 in ?? ()
> #113 0x081521b0 in ?? ()
> #114 0xfffffffe in ?? ()
> #115 0xbfffce98 in ?? ()
> #116 0x417a1a7b in ?? ()
> #117 0x081523b8 in ?? ()
> #118 0x00000001 in ?? ()
> #119 0xbfffce8c in ?? ()
> #120 0x417b6734 in ?? ()
> #121 0x00000000 in ?? ()
> #122 0xfffffffe in ?? ()
> #123 0xbfffcee8 in ?? ()
> #124 0x4178dda5 in ?? ()
> #125 0x081520f0 in ?? ()
> #126 0x00000002 in ?? ()
> #127 0x00000000 in ?? ()
> #128 0x081523b8 in ?? ()
> #129 0xbfffcecc in ?? ()
> #130 0x08151c30 in ?? ()
> #131 0x00000000 in ?? ()
> #132 0x00000000 in ?? ()
> #133 0x417b6240 in ?? ()
> #134 0x00000000 in ?? ()
> #135 0x00000000 in ?? ()
> #136 0x00000000 in ?? ()
> #137 0x00000000 in ?? ()
> #138 0x00000000 in ?? ()
> #139 0x00000002 in ?? ()
> #140 0x41776888 in ?? ()
> #141 0x00000001 in ?? ()
> #142 0x08151c20 in ?? ()
> #143 0xbfffcf38 in ?? ()
> #144 0x4176a026 in ?? ()
> #145 0x417760c8 in ?? ()
> #146 0x00000002 in ?? ()
> #147 0x00000000 in ?? ()
> #148 0x00000000 in ?? ()
> #149 0x08151c30 in ?? ()
> #150 0xbfffcf10 in ?? ()
> #151 0x41777b00 in ?? ()
> #152 0x00000000 in ?? ()
> #153 0x08151c30 in ?? ()
> #154 0x00000000 in ?? ()
> #155 0x00000000 in ?? ()
> #156 0x00000064 in ?? ()
> #157 0x00000000 in ?? ()
> #158 0x00000000 in ?? ()
> #159 0x00000000 in ?? ()
> #160 0x41776888 in ?? ()
> #161 0x00000001 in ?? ()
> #162 0x08151c20 in ?? ()
> #163 0xbfffcf68 in ?? ()
> #164 0x4176a552 in ?? ()
> #165 0x08152438 in ?? ()
> #166 0x08151c20 in ?? ()
> #167 0x4043eb40 in timezone () from /lib/libc.so.6
> #168 0x000007fa in ?? ()
> #169 0x00000000 in ?? ()
> #170 0x00000000 in ?? ()
> #171 0x41776888 in ?? ()
> #172 0x41777b94 in ?? ()
> #173 0x4176cfc0 in ?? ()
> #174 0xbfffcfa8 in ?? ()
> #175 0x4176b093 in ?? ()
> #176 0x08151c20 in ?? ()
> #177 0x4043eb40 in timezone () from /lib/libc.so.6
> #178 0x00000800 in ?? ()
> #179 0x4043dc80 in ?? () from /lib/libc.so.6
> #180 0x4176cfc0 in ?? ()
> #181 0x00000000 in ?? ()
> #182 0x00000000 in ?? ()
> #183 0x00000000 in ?? ()
> #184 0x00000000 in ?? ()
> #185 0x00000000 in ?? ()
> #186 0x41776888 in ?? ()
> #187 0x00000000 in ?? ()
> #188 0x00000000 in ?? ()
> #189 0xbfffcfe8 in ?? ()
> #190 0x4176b016 in ?? ()
> #191 0x00000000 in ?? ()
> #192 0x41777b94 in ?? ()
> #193 0x4043eb40 in timezone () from /lib/libc.so.6
> #194 0x00000800 in ?? ()
> #195 0x4043dc80 in ?? () from /lib/libc.so.6
> #196 0x4177d740 in ?? ()
> #197 0x00000002 in ?? ()
> #198 0x00000000 in ?? ()
> #199 0x4176cfc0 in ?? ()
> #200 0x4043ebc0 in timezone () from /lib/libc.so.6
> #201 0x41776888 in ?? ()
> #202 0x4043f844 in _res () from /lib/libc.so.6
> #203 0xbfffd018 in ?? ()
> #204 0x4176d973 in ?? ()
> #209 0x4177d740 in ?? ()
> #210 0x00000002 in ?? ()
> #211 0x4176cfc0 in ?? ()
> #212 0x403f7f89 in __h_errno_location () from /lib/libc.so.6
> /usr/lib/cyrus/get-backtrace.gdb:2: Error in sourced command file:
> Previous frame inner to this frame (corrupt stack?)
>
> i don't think i have memory problems, kernel logs don't show anything 
> about that, the program memtest run without problems, i once have a 
> problem with the hardware clock, and i keep the box syncronizing every 
> minute since about 3 months ago.
>
> SASL seems to be ok, because cyrus can authenticate well, when i 
> authenticate with a wrong password it show a login failure:
>
> Login FAILED:
>
> root at pascacio:#  imtest -a ndiaz localhost
> S: * OK pascacio Cyrus IMAP4 v2.2.13-Debian-2.2.13-8~bpo1 server ready
> C: C01 CAPABILITY
> S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS 
> NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UN
> SELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT 
> THREAD=REFERENCES ANNOTATEMORE IDLE
> S: C01 OK Completed
> Please enter your password:
> C: L01 LOGIN ndiaz {8}
> S: + go ahead
> C: <omitted>
> S: L01 NO Login failed: authentication failure
> Authentication failed. generic failure
> Security strength factor: 0
> C: Q01 LOGOUT
> Connection closed.
>
> Login OK:
>
> root at pascacio# imtest -a ndiaz localhost
> S: * OK pascacio Cyrus IMAP4 v2.2.13-Debian-2.2.13-8~bpo1 server ready
> C: C01 CAPABILITY
> S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS 
> NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UN
> SELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT 
> THREAD=REFERENCES ANNOTATEMORE IDLE
> S: C01 OK Completed
> Please enter your password:
> C: L01 LOGIN ndiaz {7}
> S: + go ahead
> C: <omitted>
> failure: prot layer failure
>
> Thanks for your time, any help will be apreciated.
>


-- 
Nestor A. Diaz
Ingeniero de Sistemas
Tel. +57 1-600-5490 x 211
Cel. +57 316-227-3593
Tel. SIP: sip:211 at tiendalinux.com
Email/MSN: nestor at tiendalinux.com
http://www.tiendalinux.com/
Bogota, Colombia