Some messages are missing and reconstruct doesn't help

[Geoff Adams]

I have recently upgraded from Cyrus imapd 2.3.16 to 2.4.16, and it would seem I have a subtle problem. One user noticed that he didn't receive any new mail since the upgrade. Sure enough, I found that, while he had 16322 message files in his inbox (*. files) and growing as new mail came in, selecting his inbox only showed the constant:

* 3743 EXISTS

No incantation of reconstruct that I found would do the trick. Eventually, I had to kill the master, remove the cyrus.index (and cyrus.cache; not sure if that mattered) files, and run reconstruct on that mailbox. Now, it shows:

* 16322 EXISTS

This is great, except that I suppose now he's lost all his seen state and flag settings.

Here are the before and after index files:

-rw-------   1 cyrus    mail     25036324 Aug 24 03:54 cyrus.cache
-rw-------   1 cyrus    mail     25036324 Aug 23 03:44 cyrus.cache,old
-rw-------   1 cyrus    mail         174 Aug 24 03:52 cyrus.header
-rw-------   1 cyrus    mail     1567040 Aug 24 03:54 cyrus.index
-rw-------   1 cyrus    mail     1567040 Aug 24 03:44 cyrus.index,old

Notice that the size didn't change. And yet cmp shows that the contents differ as soon as the third byte. I don't know the file format enough to dive into it looking for the corruption, of course...

After checking, I've noticed that a number of other user mailboxes also show a fraction of the messages in their corresponding mailbox directories. It seems that the missing messages are spread throughout the mailboxes, though, so no one seems to have noticed, yet. I checked my own mailbox, and I see the same thing. Just by looking at their content, I'm pretty sure I haven't deleted the missing messages in question, so it shouldn't be a delayed-expunge issue. I haven't expunged my inbox for a long time, anyway.

I also note that a UID FETCH for a missing message doesn't return any results.

. UID FETCH 3287 all
. OK Completed (0.020 sec)

So, what's going on, here, and why doesn't 2.4.16's reconstruct find the missing messages? (I've tried the -G, -R, and -f flags, at least.) All it does is print the mailbox name to stdout and log a message to syslog saying it's reconstructing the mailbox. No errors or other information, and no change in apparent mailbox content.

Thanks for any clues,
- Geoff