cyrus imap hangs regularly

Fri Aug 22 15:33:31 EDT 2003

Hey Guys,

These locker errors and other DB related errors continue to pop up
regularly.  Its gotten to the point where if I dont restart cyrus every 2 -
3 hours mail starts to bounce..

Another issue I noticed is that my /var/spool/imap/stage. is larger than I
used to see it and it continues to grow slowly.  Its at around 165M now.

I could be wrong but it looks like a DB corruption issue.  I've tried
using reconstruct, but it doesnt seem to help.  Not really sure how to
continue from here.  If anyone has any ideas I'd apprecaite the help.

Thanks in advance,
Ilan

Errors I commonly see now:
Aug 22 20:21:23 mail1 cvt_cyrusdb[30983]: DBERROR db4:
/var/lib/imap/user/k/kathy^pressaid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[30983]: DBERROR: opening
/var/lib/imap/user/k/kathy^pressaid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[30983]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[30983]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[30992]: DBERROR db4:
/var/lib/imap/user/n/nella^pressaid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[30992]: DBERROR: opening
/var/lib/imap/user/n/nella^pressaid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[30992]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[30992]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31001]: DBERROR db4:
/var/lib/imap/user/r/rodeo^pressaid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[31001]: DBERROR: opening
/var/lib/imap/user/r/rodeo^pressaid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31001]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[31001]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31010]: DBERROR db4:
/var/lib/imap/user/r/ruthie.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[31010]: DBERROR: opening
/var/lib/imap/user/r/ruthie.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31010]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[31010]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31019]: DBERROR db4:
/var/lib/imap/user/s/sol^pressaid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[31019]: DBERROR: opening
/var/lib/imap/user/s/sol^pressaid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31019]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[31019]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31028]: DBERROR db4:
/var/lib/imap/user/t/tommy^pressaid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[31028]: DBERROR: opening
/var/lib/imap/user/t/tommy^pressaid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31028]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[31028]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31037]: DBERROR db4:
/var/lib/imap/user/t/TOMMY^PRESSAid^com.sub: unexpected file type or format
Aug 22 20:21:23 mail1 cvt_cyrusdb[31037]: DBERROR: opening
/var/lib/imap/user/t/TOMMY^PRESSAid^com.sub: Invalid argument
Aug 22 20:21:23 mail1 cvt_cyrusdb[31037]: DBERROR db4: Database handles
open during environment close
Aug 22 20:21:23 mail1 cvt_cyrusdb[31037]: DBERROR: error exiting
application: Invalid argument
Aug 22 20:22:03 mail1 lmtpd[31093]: DBERROR: error fetching
<LISTMANAGERSQL-18042625-68240-2003.08.21-16.00.02--support#pressaid.com at bc07.standardservices.com>:
DB_PAGE_NOTFOUND: Requested page not found

------------------------------------
Ilan Rabinovitch 
ilan at fonz.net E-MAIL

The Southern California Llnux Expo: "Bringing businesses, academic
institutions and the Linux community together in a way that no other
conference does!" -- http://www.socallinuxexpo.org

On Mon, Aug 18, 2003 at 11:06:15AM -0700, Ilan Rabinovitch wrote:
> Rob,
> 
> Thanks for pointing me at those docs.  I had it going hourly but I guess
> every half hour is the way to go because its improved significantly.
> 
> I dont think low memory is the issue.  Free always reports plenty of free
> physical ram and swap.  I've been watching the logs closely lately to find
> out what kills cyrus and it seems this is the recurring error:
> Aug 18 18:49:53 imapd[3695]: DBERROR db4: 241 lockers
> 
> Any ideas are greatly appreciated.
> Thanks,
> Ilan
> 
> 
> 
> Cyrus and lmtpd seem to go to sleep and just not wake up.  This make its
> impossible for sendmail to deliver mail from its queue.
> 
> Thanks again for you help,
> Ilan
> 
> 
> On Tue, Aug 12, 2003 at 04:47:55PM -0400, Rob Siemborski wrote:
> > On Tue, 12 Aug 2003, Ilan Rabinovitch wrote:
> > 
> > > Lately I'm finding that the various cyrus processes go to "sleep" and never
> > > wake up.  As a result sendmail is unable to deliver its mail to cyrus.  This
> > > has happened 3 times in the last 48 hours.   When this happens users can connect to pop but not
> > > IMAP, but again no new mail gets delivered from the MTA.
> > 
> > This can happen if processes that are otherwise doing nothing get killed
> > unexpectedly, either by an administrative action or by a low memory
> > condition or similar.
> > 
> > > If I restart Cyrus it now takes over an hour to fully start.  It spends this
> > > time: converting db files, ctl_cyrusdb -r, and ctl_deliver -r.  While these
> > > are running users cannot connect to imap or pop.  The cyrus processes use
> > > 97% CPU on a pretty beefy machine for their entire duration.  When it completes they can
> > > connect to both just fine and I can force sendmail to deliver all the mail
> > > no problem.  However, this problem repeats just a few hours late.
> > 
> > Please read:
> > 
> > http://asg.web.cmu.edu/twiki/bin/view/Cyrus/WhyBerkeleyRecoverySoLong
> > 
> > -Rob
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> > Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456
> > Research Systems Programmer * /usr/contributed Gatekeeper
> 
> -- 
> Ilan Rabinovitch 
> ilan at fonz.net E-MAIL
> 
> The Southern California Llnux Expo: "Bringing businesses, academic institutions and the Linux community together in a way that no other conference does!" -- http://www.socallinuxexpo.org

-- 
Ilan Rabinovitch 
ilan at fonz.net E-MAIL

The Southern California Llnux Expo: "Bringing businesses, academic institutions and the Linux community together in a way that no other conference does!" -- http://www.socallinuxexpo.org