cyr_expire (2.3.16) problem

[Julien Coloos]

Hi,

Le 13/02/2013 14:02, Frank Elsner a écrit :
> Hello,
>
> from time to time we have this type of problem during the nightly cyr_expire:
>
> Feb 10 04:01:50 mailbackend-4 cyrus-backend/cyr_expire[15569]: IOERROR: user.<username> zero index/expunge record 299/301
> Feb 10 04:01:50 mailbackend-4 cyrus-backend/cyr_expire[15569]: failure expiring user.<username>: System I/O error
> Feb 11 04:01:48 mailbackend-4 cyrus-backend/cyr_expire[2010]: IOERROR: user.<username> zero index/expunge record 299/301
> Feb 11 04:01:48 mailbackend-4 cyrus-backend/cyr_expire[2010]: failure expiring user.<username>: System I/O error
> Feb 12 04:02:19 mailbackend-4 cyrus-backend/cyr_expire[27580]: IOERROR: user.<username> zero index/expunge record 299/301
> Feb 12 04:02:19 mailbackend-4 cyrus-backend/cyr_expire[27580]: failure expiring user.<username>: System I/O error
> Feb 13 04:03:15 mailbackend-4 cyrus-backend/cyr_expire[19753]: IOERROR: user.<username> zero index/expunge record 299/301
> Feb 13 04:03:15 mailbackend-4 cyrus-backend/cyr_expire[19753]: failure expiring user.<username>: System I/O error
>
> This leaves cyrus.*.NEW files in the mailbox directory of the user and
> requires a reconstruct.
>
> What's behind? And how can we avoid this problem?
If you got into a situation like ours, you may have reached the limit 
for the number of opened file descriptors at some point on your server. 
If so you may find a bunch of errors at that time in your logs; there 
should also be some of the messages mentioning "Too many open files in 
system" or similar.
In such a situation, there is a bug that can lead to empty (zeros) index 
records being added when a message is received, which later triggers 
"zero index/expunge record" errors.
If that's the case, you may try to find how many cyrus processes can run 
at the same time (maxchild of each service) and how many file 
descriptors each usually consume to determine a "safe" limit on your 
system. Note that each shared library consume one file descriptor (mmap 
by the run-time linker apparently).

I believe this situation should not happen anymore in cyrus 2.4 (there 
is an assertion preventing that). But I don't know what happens for 
those empty records upon migrating from cyrus 2.3 to 2.4.


Regards,
Julien