SELECT / Squirrelmail issue

[Carsey, Robert]

Ok.. after several hours of testing.. I think I've come to a conclusion:

Reconstruct does not handle the case of a missing cyrus.index file well.  It aborts around mailbox.c:1517 and causes problems (I think the CRC contained in cyrus.header is valid.. but the contents of cyrus.index are messed up somehow).

Below I demonstrate the creation of a new mailbox, manually add messages to it, reconstruct, its OK.    Delete cyrus.index  and reconstruct:  imapd is not happy.

Unforunately when I upgraded to 2.4.0 and ran a background index upgrade to upgrade indexes before users logged in,  some indexes were corrupt (and probably have been for a long time) and my upgrade threw some "cant mmap file" type errors.  As a result, I ended up just deleting their cyrus.index and running a reconstruct.. which seemed to work.. but as I'm finding out now, not really.

Create new mailbox:

localhost.localdomain> cm user.zhu

[root at hawkmail1 zhu]# telnet localhost 143
--cut--
. select INBOX
* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UIDVALIDITY 1287334552] Ok
* OK [UIDNEXT 1] Ok
* OK [HIGHESTMODSEQ 0] Ok
* OK [URLMECH INTERNAL] Ok
. OK [READ-WRITE] Completed

Lets add some messages to the mailbox.

[root at hawkmail1 zhu]# cp -a /tmp/[1-2]. /var/spool/imap3/K/user/zhu
[root at hawkmail1 zhu]# reconstruct user.zhu
user.zhu uid 1 found - adding
user.zhu uid 2 found - adding
user.zhu

Check again just to make sure its still happy:

[root at hawkmail1 zhu]# telnet localhost 143
--cut--
. select INBOX
* 2 EXISTS
* 2 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1287334552] Ok
* OK [UIDNEXT 3] Ok
* OK [HIGHESTMODSEQ 1] Ok
* OK [URLMECH INTERNAL] Ok
. OK [READ-WRITE] Completed

Good.  (if I reconstruct and check again, its still OK.. no issues)

Lets delete cyrus.index.  Maybe we want to delete the index because its corrupt or we want to start fresh for some reason.

[root at hawkmail1 zhu]# rm cyrus.index

Lets reconstruct.  I am using a re-compiled version of reconstruct which outputs some extra debug stuff (below).

 [root at hawkmail1 zhu]# /usr/local/src/cyrus-imapd-2.4.0/imap/reconstruct user.zhu
user.zhu: failed to read index header
user.zhu uid 1 found - adding
user.zhu uid 2 found - adding
Aborted

I added extra syslog() lines to mailbox.c:1517  to find out why it was aborting (trying to close a dirty mailbox).  I can give you a backtrace from gdb if you wish.

Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR: opening index user.zhu: No such file or directory
Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR: index record 1 for user.zhu past end of file
Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR-CUSTOM: trying to unlock a dirty mailbox: user.zhu
Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR-CUSTOM: i dirty
Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR-CUSTOM: modseq dirty
Oct 17 13:08:13 hawkmail1 reconstruct[5241]: IOERROR-CUSTOM: quota dirty

Hmm doesn't sound good.  Lets see what imap server has to say.

[root at hawkmail1 zhu]# telnet localhost 143
--cut--
. select INBOX
* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UIDVALIDITY 0] Ok
* OK [UIDNEXT 1] Ok
* OK [HIGHESTMODSEQ 0] Ok
* OK [URLMECH INTERNAL] Ok
. OK [READ-WRITE] Completed

UIDVALIDITY is not good.  (And messages do not appear in Squirrelmail).

What happens if I copy back the original cyrus.* files that were created when I created the mailbox?
 
[root at hawkmail1 zhu]# cp -a /tmp/cyrus.* .
[root at hawkmail1 zhu]# telnet localhost 143
--cut--
. select INBOX
* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UIDVALIDITY 1287336639] Ok
* OK [UIDNEXT 1] Ok
* OK [HIGHESTMODSEQ 0] Ok
* OK [URLMECH INTERNAL] Ok
. OK [READ-WRITE] Completed

Looks good again.  And if I reconstruct (to index the two messages I added since this backup copy of cyrus.*)?

Its fine.

So if the problem is as I think it is.. im not sure what the quickest way would be to fix users when they call to complain.. I don't want to delete their account, recreate it, copy the mail back in, re-add all their folders, etc.