Some messages are missing and reconstruct doesn't help

[Geoff Adams]

On Aug 24, 2012, at 11:56 PM, Bron Gondwana wrote:

> On Sat, Aug 25, 2012, at 07:28 AM, Geoff Adams wrote:
>> On Aug 24, 2012, at 3:22 PM, Geoff Adams wrote:
>> 
>>> * 3743 EXISTS
>> 
>> I think I may have solved the problem. I created a new mailbox for the user, used an IMAP client to copy the original 3743 messages into that mailbox, and then used a one-line shell script to convert the newer messages into mbox format, imported that into the mail client, and then copied those messages into the new mailbox via IMAP, as well.
>> 
>> I had to go through mbox format because I couldn't figure out a way to have reconstruct use the message file timestamps for the internaldate; all the new messages were timestamped as the moment I did the reconstruct. Am I missing something, here? That seems like an obvious and trivial feature.
> 
> On 2.4.x it should be using the timestamp of the message file.

It really isn't, though. Worse, without the -R flag, I think it is, reconstruct will touch all the message files, changing their date to the time reconstruct was run. Not really nice.

Is there some option I'd need to turn on to enable using the timestamp for internaldate?

> Actually, if you enable internaldate_heuristic in imapd.conf, it will use the
> date of the most recent "Received" header instead, which is generally really
> good, since it gets created by Cyrus during the LMTP delivery.  Just not so
> handy for Sent mail…

Ah, good to know!

>> I then swapped the new mailbox in as the inbox, and reconstructed, to make sure the mailbox uids are in sync. So far, so good. New messages are delivered, and actually show up in the IMAP client. As do all 3836 messages, including the ones both before and after the upgrade.
>> 
>> But I'm still curious what corrupt state that mailbox was in, and how it got there. It seems related to the 2.3.x -> 2.4.x upgrade, since that's the line beyond which new messages disappear.
> 
> Me too.  There wasn't anything from the cyrus.index files to suggest corruption.

Unfortunately, it's happened again.

That same user is now unable to receive any new messages in his inbox. After following the procedure I described above, I was able to send a message to him, and it appeared in his inbox. Since then, another 16 messages have been delivered, and the message files are in the inbox, but they do not appear via IMAP, nor does reconstruct see them. I'm at a loss.

I've moved that mailbox aside, and just re-created his inbox using cyradm. I hope that finally allows this user to receive new mail. I suspect, though, that if he or I were to drag his old mail back into his new inbox, the problem would re-appear (since that's basically what I did last time). It seems like something in one of the mail messages is leading to corruption in the index, although that seems improbable.

?!

- Geoff