LMTP, db4, cyrus memory leak?

[Joe Hrbek]

Ok, so I have a really strange problem.  It appears to me that cyrus or
saslauthd has a memory leak of some kind, and if not those two, something
else in the chain.

My mail system is setup a little different from most.  I am using:

p3 dual 500, 512mb ram
Cyrus IMAP4 v2.1.9-Invoca-RPM-2.1.9-4
Redhat 8.0 w/ their sasl stuff
Postfix 1.10
mysql 3.23
bdb4

I had to rebuild the cyrus rpm so it would install using db4 instead of db3,
but other than that, nothing wierd in the system.

What is bothering me is this error in particular (or any that resemble it):

Oct 29 09:13:36 mail lmtpd[10260]: DBERROR db4: 9 lockers

Although this message is explained in:

http://www.irbs.net/internet/info-cyrus/0202/0432.html

It still doesn't explain why my system is running out of memory. :(  I have
noticed that if I restart saslauthd I get a considerable amount of memory
back.  If I restart cyrus I get a bit more.  If I don't restart these two
services, after about 4 days, the system starts killing processes as
described below.

Oct 29 09:13:36 mail lmtpd[10260]: DBERROR db4: 9 lockers
Oct 29 09:13:36 mail lmtpd[10256]: DBERROR db4: 10 lockers
Oct 29 09:13:37 mail lmtpd[10321]: DBERROR db4: 12 lockers
Oct 29 09:13:38 mail lmtpd[10373]: DBERROR db4: 13 lockers
Oct 29 09:13:38 mail pop3d[10230]: login: [192.168.44.65]
sdrudge.gplsinc.com plaintext
Oct 29 09:13:38 mail lmtpd[10268]: DBERROR db4: 14 lockers
Oct 29 09:13:45 mail lmtpd[10388]: DBERROR db4: 20 lockers
Oct 29 09:13:45 mail lmtpd[10256]: DBERROR db4: 18 lockers
Oct 29 09:13:45 mail lmtpd[10298]: DBERROR db4: 18 lockers
Oct 29 09:13:45 mail lmtpd[10368]: DBERROR db4: 18 lockers
Oct 29 09:13:45 mail lmtpd[10260]: DBERROR db4: 18 lockers
Oct 29 09:13:45 mail lmtpd[10321]: DBERROR db4: 18 lockers
Oct 29 09:13:45 mail lmtpd[10393]: DBERROR db4: 20 lockers
Oct 29 09:13:45 mail lmtpd[10268]: DBERROR db4: 21 lockers
Oct 29 09:13:49 mail lmtpd[10256]: DBERROR db4: 15 lockers
Oct 29 09:13:49 mail lmtpd[10388]: DBERROR db4: 15 lockers
Oct 29 09:13:49 mail lmtpd[10373]: DBERROR db4: 15 lockers
Oct 29 09:13:49 mail lmtpd[10391]: DBERROR db4: 15 lockers
Oct 29 09:13:50 mail lmtpd[10409]: DBERROR db4: 18 lockers
Oct 29 09:13:51 mail lmtpd[10409]: DBERROR db4: 20 lockers
Oct 29 09:13:52 mail lmtpd[10368]: DBERROR db4: 19 lockers
Oct 29 09:13:52 mail lmtpd[10298]: DBERROR db4: 19 lockers
Oct 29 09:14:32 mail lmtpd[10409]: DBERROR db4: 18 lockers
Oct 29 09:14:45 mail master[1681]: process 10256 exited, signaled to death
by 9
Oct 29 09:14:45 mail kernel: Out of Memory: Killed process 10256 (lmtpd).
Oct 29 09:14:59 mail lmtpd[10393]: DBERROR db4: 18 lockers
Oct 29 09:14:59 mail lmtpd[10368]: DBERROR db4: 18 lockers
Oct 29 09:14:59 mail lmtpd[10388]: DBERROR db4: 18 lockers
Oct 29 09:15:00 mail lmtpd[10400]: DBERROR db4: 18 lockers
Oct 29 09:15:10 mail kernel: Out of Memory: Killed process 10368 (lmtpd).
Oct 29 09:15:11 mail master[1681]: process 10368 exited, signaled to death
by 9
Oct 29 09:15:32 mail kernel: Out of Memory: Killed process 10260 (lmtpd).
Oct 29 09:15:32 mail master[1681]: process 10260 exited, signaled to death
by 9
Oct 29 09:15:55 mail master[1681]: process 10298 exited, signaled to death
by 9
Oct 29 09:15:54 mail kernel: Out of Memory: Killed process 10298 (lmtpd).
Oct 29 09:15:58 mail lmtpd[10409]: DBERROR db4: 21 lockers
Oct 29 09:15:59 mail lmtpd[10373]: DBERROR db4: 21 lockers
Oct 29 09:15:59 mail lmtpd[10391]: DBERROR db4: 21 lockers
Oct 29 09:16:06 mail kernel: Out of Memory: Killed process 10268 (lmtpd).
Oct 29 09:16:06 mail master[1681]: process 10268 exited, signaled to death
by 9
Oct 29 09:16:16 mail lmtpd[10321]: DBERROR db4: 21 lockers
Oct 29 09:16:31 mail kernel: Out of Memory: Killed process 10321 (lmtpd).
Oct 29 09:16:39 mail master[1681]: process 10321 exited, signaled to death
by 9
Oct 29 09:16:42 mail pop3d[10126]: login: [192.168.0.71] cpeck.gplsinc.com
plaintext
Oct 29 09:17:01 mail kernel: Out of Memory: Killed process 10373 (lmtpd).
Oct 29 09:17:01 mail master[1681]: process 10373 exited, signaled to death
by 9
Oct 29 09:17:05 mail pop3d[8629]: login: [192.168.14.74] gklein.gplsinc.com
plaintext
Oct 29 09:17:05 mail pop3d[8877]: login: [192.168.25.69]
lstuenkel.gplsinc.com plaintext
Oct 29 09:17:10 mail pop3d[10230]: login: [192.168.14.61]
smartin.gplsinc.com plaintext
Oct 29 09:17:10 mail pop3d[8723]: login: [192.168.0.54] jallison.gplsinc.com
plaintext
Oct 29 09:17:26 mail kernel: Out of Memory: Killed process 10388 (lmtpd).
Oct 29 09:17:48 mail last message repeated 4 times
Oct 29 09:17:52 mail master[1681]: process 10388 exited, signaled to death
by 9
Oct 29 09:17:55 mail master[1681]: process 10391 exited, signaled to death
by 9
Oct 29 09:17:55 mail kernel: Out of Memory: Killed process 10391 (lmtpd).
Oct 29 09:17:59 mail pop3d[10261]: login: [192.168.14.66] hbohac.gplsinc.com
plaintext
Oct 29 09:17:59 mail pop3d[10429]: login: [192.168.14.64]
ngaskill.gplsinc.com plaintext
Oct 29 09:17:59 mail pop3d[9787]: login: [192.168.15.53]
tcarnahan.gplsinc.com plaintext
Oct 29 09:18:07 mail kernel: Out of Memory: Killed process 10393 (lmtpd).
Oct 29 09:18:07 mail master[1681]: process 10393 exited, signaled to death
by 9
Oct 29 09:18:07 mail pop3d[8953]: login: [192.168.0.60] dbaker.gplsinc.com
plaintext
Oct 29 09:18:07 mail pop3d[10396]: login: [192.168.44.56]
ccleary.gplsinc.com plaintext
Oct 29 09:18:07 mail pop3d[10441]: login: [192.168.14.68]
dkenyon.gplsinc.com plaintext
Oct 29 09:18:17 mail master[1681]: process 10409 exited, signaled to death
by 9
Oct 29 09:18:17 mail kernel: Out of Memory: Killed process 10409 (lmtpd).
Oct 29 09:18:38 mail pop3d[10437]: login: [192.168.44.50]
josborne.gplsinc.com plaintext
Oct 29 09:18:38 mail pop3d[10433]: login: [192.168.34.51]
jmorris.gplsinc.com plaintext
Oct 29 09:19:00 mail kernel: Out of Memory: Killed process 10400 (lmtpd).
Oct 29 09:19:00 mail master[1681]: process 10400 exited, signaled to death
by 9
Oct 29 09:19:10 mail master[1681]: process 10416 exited, signaled to death
by 9
Oct 29 09:19:10 mail kernel: Out of Memory: Killed process 10416 (lmtpd).
Oct 29 09:19:19 mail kernel: Out of Memory: Killed process 10417 (lmtpd).
Oct 29 09:19:42 mail last message repeated 3 times
Oct 29 09:19:42 mail master[1681]: process 10417 exited, signaled to death
by 9
Oct 29 09:19:56 mail kernel: Out of Memory: Killed process 10418 (lmtpd).
Oct 29 09:19:59 mail master[1681]: process 10418 exited, signaled to death
by 9
Oct 29 09:20:06 mail kernel: Out of Memory: Killed process 10427 (lmtpd).
Oct 29 09:20:19 mail master[1681]: process 10427 exited, signaled to death
by 9
Oct 29 09:20:24 mail master[1681]: process 10454 exited, signaled to death
by 9
Oct 29 09:20:24 mail kernel: Out of Memory: Killed process 10454 (lmtpd).
Oct 29 09:20:29 mail ctl_cyrusdb[10509]: checkpointing cyrus databases
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 1549 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 1551 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 1552 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 9772 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 9773 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10092 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10286 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10287 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10338 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10339 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10472 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10478 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10481 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10495 (mysqld).
Oct 29 09:20:40 mail kernel: Out of Memory: Killed process 10503 (mysqld).
Oct 29 09:20:56 mail saslauthd[1618]: Lost connection to MySQL server during
query
Oct 29 09:20:56 mail saslauthd[1624]: Lost connection to MySQL server during
query
Oct 29 09:20:56 mail saslauthd[1624]: AUTHFAIL: user=mspangler.gplsinc.com
service=pop realm= [PAM auth error]
Oct 29 09:20:56 mail saslauthd[1618]: AUTHFAIL: user=plarkin.gplsinc.com
service=pop realm= [PAM auth error]
....etc...etc...


Adding more memory will only massage the issue here that it appears that the
the SASLauthd or cyrus IMAP has a memory leak.

This is really what killed the system though.

Oct 29 09:14:45 mail master[1681]: process 10256 exited, signaled to death
by 9
Oct 29 09:15:11 mail master[1681]: process 10368 exited, signaled to death
by 9
Oct 29 09:15:32 mail master[1681]: process 10260 exited, signaled to death
by 9
....etc...etc...

The kernel just started dropping processes. :(


So, to fix this with a big band-aid, I have a daily cron job restarting
saslauthd and cyrus everyday at midnight.  I would like to fix this properly
though, so if anyone has any ideas, I would love to hear them. :)

Thanks!

-Joe Hrbek