Some messages are missing and reconstruct doesn't help

[Geoff Adams]

So, based on your expert investigation, I'm now going on the theory that most of the "missing" messages are, in fact, expunged, and only the newer messages since the upgrade are missing. I can correct that…

I've killed the master, copied the cyrus.* files for that mailbox from the pre-upgrade snapshot, and then re-run reconstruct on that mailbox. That took a while, and seemed successful, finding dozens of missing messages, as I expected--all the messages delivered since the upgrade.

I now see a small number of messages in the inbox, but more than the 3743 messages visible before:

* 3836 EXISTS

I think that corresponds to the 3743 non-expunged, pre-upgrade messages plus the newly-delivered messages.

However, after connecting to it using the Apple Mail client (which, as far as I know, just iterates over the mailbox, caching whatever data it cares about for all the messages), we're right back to where we were:

* 3743 EXISTS

I find it interesting that message 3743 is UID 23400, which has an older internaldate than message 3742, uid 23396, which is the latest message visible once again in this mailbox, and just about the last message delivered before I upgraded.

. fetch 3742:3743 (uid internaldate)
* 3742 FETCH (UID 23396 INTERNALDATE "16-Aug-2012 06:47:20 -0400")
* 3743 FETCH (UID 23400 INTERNALDATE "12-Aug-2012 14:16:51 -0400")
. OK Completed (0.010 sec)

Message 3743 would probably have been one of these:

-rw-------   3 cyrus    mail        4928 Aug 12 14:16 23400.
...
-rw-------   1 cyrus    mail       14394 Aug 16 06:47 23396.
-rw-------   2 cyrus    mail       36194 Aug 16 07:28 23397.
-rw-------   2 cyrus    mail       38333 Aug 16 10:08 23398.
-rw-------   2 cyrus    mail       27516 Aug 16 10:10 23399.
-rw-------   1 cyrus    mail        3196 Aug 16 12:04 23401.

It seems perfectly valid for the internaldates not to be in uid order, due to refiling. But could that confuse something in the indexing code?

I'm at a loss, here.

- Geoff