IOERROR: zero index/expunge record

[Bron Gondwana]

On Wed, 03 Aug 2011 23:13:16 +0200, Marc Muehlfeld  
<Marc.Muehlfeld at medizinische-genetik.de> wrote:

> Am 03.08.2011 22:37, schrieb Bron Gondwana:
>> And let's have copies of your imapd.conf and cyrus.conf while
>> we're at it.
>
> /etc/imapd.conf:
> http://pastebin.com/D8Szgmqk
>
> /etc/cyrus.conf:
> http://pastebin.com/CL2M64AR

Sure.

>> So you have corrupted cyrus.index files. I don't suppose you kept any
>> of them?
>
> I can send you a copy of a complete broken mail folder including  
> cyrus.cache, cyrus.cache.NEW, cyrus.header, cyrus.index, cyrus.index.NEW  
> and mails. It's nothing secret. It's a subfolder of my mailbox where I  
> put the mails from this mailing list in. I'll send it in a separate mail  
> to your address.

Thanks, comments below.

>> Do you have delayed expunge?
>
> Where can I check this?

imapd.conf, and no you don't.

>> I do recommend upgrading to 2.4.10, despite the initial IO hit of the
>> upgrade. The repack on expunge in 2.3.x was really, really complex
>> and buggy.
>
> If there is a change, we would stay at the version that comes with the  
> distrubution, because then we can easily update on bugs or security  
> issues through the repository. But if there's a bug, that could be fixed  
> I, or whoever, can send it to redhat.

Good luck with that.  It's not an easy fix, which is part of why I rewrote
that whole section after a couple of years of soul searching about the
amount of work it would bloody well entail.  Took about a year.

Here's some interesting dump from your cyrus.index:

23 1311668694 1311674400 6022 5038 5038 40660 1311668975  
00000000000000000000000000000000  
00000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000  
34 2 0000000000000000000000000000000000000000 1
0 0 0 0 0 0 0 0 00000000000000000000000000000000  
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000  
0 0 0000000000000000000000000000000000000000 0
25 1311688444 1311674400 5130 4545 4545 45252 1311692683  
00000000000000000000000000000000  
00000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000  
18 2 0000000000000000000000000000000000000000 1

All the UIDs are perfectly sane right up until 24, which is totally blank.

Then 25 is good again.  That last number is MODSEQ - so there's no  
corruption
at all within the records.  A totally blank record got written.

It's also not anywhere "interesting" blockwise, so unlikely to be any OS
MMAP problem (we have had them).

00000840  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 00   
|................|
00000850  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   
|................|
*
000008a0  00 00 00 19 4e 2e c6 fc  4e 2e 90 20 00 00 14 0a  |....N...N..  
....|
000008b0  00 00 11 c1 00 00 11 c1  00 00 b0 c4 4e 2e d7 8b   
|............N...|


That's the hexdump of the interesting area - a long run of zero bytes.

Unsurpringly, cyrus.index.NEW runs up to the same point and then throws an  
error.

24. is a perfectly normal email file - so nothing going on there.

SOMETHING wrote a blank record.  It would be significantly easier to find  
that
something on 2.4 because there's only one API function that can write to
cyrus.index records (strictly there are two - one handles appends and the
other handles rewrites - but they both use the same supporting function to
format the record)

Now as for what did it?  That's going to be harder to track down.  How
comfortable are you with patching and building your own Cyrus?

While it's nice to run the distribution package, distributions are
shipping disgracefully old versions of Cyrus.  At least 2.3.16 isn't
too bad!

I'm afraid the next thing is to look for anything interesting in your
syslog, but changes are that there's not enough information being
logged to help.

The problem is, this zeroed record could be caused anywhere, and you
wouldn't find out until you went to repack during an expunge.

Bron.

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/