BIG PROBLEM: Need help with production box

[Curtis Robinson]

Cyrus is using a custom compiled Berkeley DB version with using the default compile options.
The berkeley db version is 4.1.25.  This problem is new.  This system has been running for monthes
with changing the software.  So, it is hard to say what the problem is.  I will try the debugging 
route, but that is terrority I am not familiar with.  I will do the ulimit on memory and see what
happens.


-----Original Message-----
From: Craig Ringer [mailto:craig at postnewspapers.com.au]
Sent: Fri 4/2/2004 3:27 AM
To: Curtis Robinson
Cc: info-cyrus at lists.andrew.cmu.edu
Subject: Re: BIG PROBLEM: Need help with production box
 
On Fri, 2004-04-02 at 15:54, Curtis Robinson wrote:
> Hello,
> 
> I need major assistance.  We have a dual cpu w/ 2GB of RAM.  Recently, lmtpd and imapd processes have been eating up all available memory to the point the machine stops responding.  I have try to reconstruct every mailbox and fix the mailbox database, but nothing has fixed it.  Some of the ctl_ commands fail too by eating up memory.  None of the users have very large mailboxes either.  Any help would be great.  Thank you.

Workaround to stop the machine from crashing: set a ulimit on user
'cyrus' such that master and it's children can't eat all available RAM.
This should at least stop your server thrashing its self to death,
though it won't solve your problem. imapds will die suddenly instead of
OOMing the machine.

Of course, then you might risk db corruption. Make  text copy of your
mailboxes.db now (ctl_mboxlist -d > text_mailboxes_list) and BE CAREFUL.

My next step would be to create another entry in cyrus.conf that ran an
imapd with debugging enabled, and an alternate config file (eg
/etc/imapd_debug.conf). I'd start with strace as the debugger, and see
if I could figure out what it was doing to cause it to eat all that RAM.

Then again, I'm not a Cyrus developer nor even a particularly
experienced user, so this may be entirely the wrong way to go about it.

example cyrus.conf entry:

	imapdebug cmd="imapd -C /etc/imapdebug.conf -D" \
                  listen="127.0.0.1:14300" prefork=0

then copy /etc/imapd.conf to /etc/imapdebug.conf and add a
'debug_command' line.

Also, as mentioned by Michael Loftis it could be a BDB issue with RH9.
In particular, the RH9 BDB is built for a different threading model, and
that can cause all sorts of fun. I've had good results with Cyrus on
RH9, but I built db4.1 in /usr/local and used that for Cyrus. Perhaps
that might be a good first step (but BACK UP YOUR MAILBOX LIST, or
preferably your entire mail system, first, and have a plain text copy of
the mailboxes.db on hand.).

Craig Ringer





---
Home Page: http://asg.web.cmu.edu/cyrus
Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html