Cyrus 2.2.8: imapd dies + corrupt inbox

[Bernd Nies]

Hi again,

>> We are running Cyrus 2.2.8 on Solaris 8 using saslauthd and LDAP 
>> authentication.
>>
>> Since we upgraded to Cyrus 2.2.8 it happens frequently that users 
>> inboxes get corrupted and clients cannot connect. Is this a known bug 
>> and has it been fixed with Cyrus 2.2.10?
>>
>> The entry in the logfile is:
>>
>> Jan 13 08:23:59 ducati master[29375]: [ID 970914 local6.error] process 
>> 16678 exited, signaled to death by 11
>> Jan 13 08:23:59 ducati master[29375]: [ID 621917 local6.debug] service 
>> imap pid 16678 in BUSY state: terminated abnormally
>>
>> A pstack on the imapd core file gives:
>>
>> core 'core' of 16678:   imapd
>>  0003b060 mailbox_expunge (1381b0, 0, 0, 16227, 112d70, 0) + 4c0
>>  00022800 cmd_expunge (119b38, 0, 119778, 107c00, 1, 11acf0) + 58
>>  0001b71c cmdloop  (0, 131db8, 0, 0, 1c68c, 0) + d94
>>  0001a6a4 service_main (0, 1134c8, ffbefa3c, 2, 0, 18dc0) + 320
>>  00018f14 main     (417ca8bf, f4598, ffbefa3c, 1130a8, 69800, 3c) + 87c
>>  00018260 _start   (0, 0, 0, 0, 0, 0) + 5c
> 
> 
> Cyrus IMAP 2.2.10 did not fix the problem with crashing imapd processes.
> I have opened a bugzilla for this
> 
>   https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2617
> 
> and hope somone might have a look at this. I can provide core files but
> I don't want to post them as bugzilla attachment since they may contain
> sensitive information.

These crashes are caused because cyrus runs out of file descriptors:


Feb 18 14:37:51 mailsrv lmtpunix[25178]: [ID 136705 local6.error]
IOERROR: opening /var/spool/adnmail/cyrus/mail/dev/a2k-dev/cyrus.cache:
Too many open files
Feb 18 14:37:51 mailsrv lmtpunix[25178]: [ID 860734 local6.debug]
verify_user(dev.a2k-dev) failed: System I/O error
Feb 18 14:38:03 mailsrv imap[546]: [ID 299345 local6.debug] SQUAT failed
to open index file
Feb 18 14:38:03 mailsrv imap[546]: [ID 418793 local6.debug] SQUAT failed
Feb 18 14:38:30 mailsrv master[19603]: [ID 970914 local6.error] process
6771 exited, signaled to death by 11
Feb 18 14:38:30 mailsv master[19603]: [ID 621917 local6.debug] service
imaps pid 6771 in BUSY state: terminated abnormally
Feb 18 14:38:43 mailsrv master[19603]: [ID 970914 local6.error] process
1326 exited, signaled to death by 11

A couple of days ago I increased the default file descriptor settings in
/etc/system from default

set rlim_fd_cur=1024
set rlim_fd_max=4096

to

set rlim_fd_cur=32768
set rlim_fd_max=65536

But recently Cyrus ran out again of file descriptors.

Tuning guides tell that it is dangerous to increase these default values:

http://docs.sun.com/source/817-6255/SystemTuning.html
http://www.princeton.edu/~psg/unix/solaris/troubleshoot/filedesc.html

We have about 100 concurrent imap users and some of them have more than
10000 mails in a folder. Postfix runs on the same host and delivers the
mails through a lmtp socket.

Could it possibly be that Cyrus may not close correctly every file after
accessing it and therefore runs out of file descriptors? Or what is the
recommended setting?

Regards,
Bernd

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html