Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

Kenneth Marshall ktm at rice.edu
Mon Jul 25 07:57:54 EDT 2016 

On Mon, Jul 25, 2016 at 03:53:43PM +1000, ellie timoney wrote:
> I don't have a 2.3.x environment handy to try to reproduce this from the
> outside in, so I'm starting at the error code and working outwards...
> 
> > > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > > >1469384362>a0080 NO Mailbox format corruption detected
> 
> "Mailbox format corruption detected" corresponds with
> IMAP_MAILBOX_CHECKSUM
> 
> This error code is exclusively returned by functions in imap/mailbox.c,
> for index/record crc mismatches.  Given that the mailbox is selectable
> and such, I'm supposing that the whole-file/whole-record crc checks are
> okay...
> 
> It looks like 2.3.16 had MAILBOX_MINOR_VERSION 10, and in particular
> doesn't have the record->cache_crc field, which many of the
> IMAP_MAILBOX_CHECKSUM checks are for.  When reading mailboxes < 12 in
> newer Cyrus versions, this field is initialised to zero.
> 
> Some of the places that want to do a crc check against record->cache_crc
> first make sure record->cache_crc is non-zero before doing so: 
> cache_append_record(), mailbox_cacherecord().
> 
> But some don't, or at least don't appear to: mailbox_append_cache()
> initially uses cache_append_record() (good), but then reads it straight
> back in to ensure freshness -- and then having done so, appears to do an
> unconditional check against record->cache_crc (dodgy?)
> 
> Actually that seems like all the spots this check happens -- so the fix
> might be as simple as the attached patch?
> 
> Kenneth, are you able to try this out?
> 
> Cheers,
> 
> ellie
> 

Hi Ellie,

I will try it this morning.

Regards,
Ken

> On Mon, Jul 25, 2016, at 02:02 PM, Bron Gondwana wrote:
> > Thanks for reporting this.  Ellie, if you get a chance can you look at
> > it?  I'm in the middle of the the FastMail security rollout right now, so
> > I can't do anything today.
> > 
> > Bron.
> > 
> > On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote:
> > > Hi,
> > > 
> > > I accidentally hijacked another thread. So here is a new message. I am
> > > testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
> > > moving messages between folders before the 'reconstruct -V max' has been
> > > run on the mailboxes. Here is the error from the IMAP client:
> > > 
> > > imap_copy_messages [a0080 NO Mailbox format corruption detected]?
> > > 
> > > and the details from the IMAP logs:
> > > 
> > > <1469384341<a0077 NOOP
> > > >1469384341>a0077 OK Completed
> > > <1469384354<a0078 LIST "" "%"
> > > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > > * LIST (\HasChildren) "/" DSPAM_notspam
> > > * LIST (\HasChildren) "/" DSPAM_spam
> > > * LIST (\HasNoChildren) "/" Drafts
> > > * LIST (\HasChildren) "/" Mail
> > > * LIST (\HasNoChildren) "/" Sent
> > > * LIST (\HasNoChildren) "/" Spam
> > > * LIST (\HasNoChildren) "/" Trash
> > > a0078 OK Completed (0.240 secs 356 calls)
> > > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> > > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > > a0079 OK Completed
> > > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > > >1469384362>a0080 NO Mailbox format corruption detected
> > > <1469384370<a0081 LIST "" "%"
> > > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > > * LIST (\HasChildren) "/" DSPAM_notspam
> > > * LIST (\HasChildren) "/" DSPAM_spam
> > > * LIST (\HasNoChildren) "/" Drafts
> > > * LIST (\HasChildren) "/" Mail
> > > * LIST (\HasNoChildren) "/" Sent
> > > * LIST (\HasNoChildren) "/" Spam
> > > * LIST (\HasNoChildren) "/" Trash
> > > a0081 OK Completed (0.230 secs 356 calls)
> > > <1469384372<a0082 CLOSE
> > > a0083 MYRIGHTS "Drafts"
> > > a0084 SELECT "Drafts"
> > > >1469384372>a0082 OK Completed
> > > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > > a0083 OK Completed
> > > >1469384372>* 0 EXISTS
> > > * 0 RECENT
> > > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> > > * OK [UIDVALIDITY 1345619846] Ok
> > > * OK [UIDNEXT 6] Ok
> > > * OK [HIGHESTMODSEQ 1] Ok
> > > * OK [URLMECH INTERNAL] Ok
> > > * OK [ANNOTATIONS 65536] Ok
> > > a0084 OK [READ-WRITE] Completed
> > > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > > * LIST (\HasChildren) "/" DSPAM_notspam
> > > * LIST (\HasChildren) "/" DSPAM_spam
> > > * LIST (\HasNoChildren) "/" Drafts
> > > * LIST (\HasChildren) "/" Mail
> > > * LIST (\HasNoChildren) "/" Sent
> > > * LIST (\HasNoChildren) "/" Spam
> > > * LIST (\HasNoChildren) "/" Trash
> > > a0078 OK Completed (0.240 secs 356 calls)
> > > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> > > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > > a0079 OK Completed
> > > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > > >1469384362>a0080 NO Mailbox format corruption detected
> > > <1469384370<a0081 LIST "" "%"
> > > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > > * LIST (\HasChildren) "/" DSPAM_notspam
> > > * LIST (\HasChildren) "/" DSPAM_spam
> > > * LIST (\HasNoChildren) "/" Drafts
> > > * LIST (\HasChildren) "/" Mail
> > > * LIST (\HasNoChildren) "/" Sent
> > > * LIST (\HasNoChildren) "/" Spam
> > > * LIST (\HasNoChildren) "/" Trash
> > > a0081 OK Completed (0.230 secs 356 calls)
> > > <1469384372<a0082 CLOSE
> > > a0083 MYRIGHTS "Drafts"
> > > a0084 SELECT "Drafts"
> > > >1469384372>a0082 OK Completed
> > > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > > a0083 OK Completed
> > > >1469384372>* 0 EXISTS
> > > * 0 RECENT
> > > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> > > * OK [UIDVALIDITY 1345619846] Ok
> > > * OK [UIDNEXT 6] Ok
> > > * OK [HIGHESTMODSEQ 1] Ok
> > > * OK [URLMECH INTERNAL] Ok
> > > * OK [ANNOTATIONS 65536] Ok
> > > a0084 OK [READ-WRITE] Completed
> > > <1469384392<a0085 NOOP
> > > >1469384392>a0085 OK Completed
> > > <1469384412<a0086 NOOP
> > > >1469384412>a0086 OK Completed
> > > 
> > > Is this a known issue with the upgrade process? Messages can be delivered to
> > > the folders via a sieve script, but both message moves and copies fail. If there
> > > is any fix for this issue, it would be greatly appreciated. I have been waiting
> > > since the release of 2.4.x for a version that would handle the I/O storm caused
> > > by automatically upgrading the mailbox format on first access, and 2.5.x does
> > > manage that successfully. Thank you for any assistance.
> > > 
> > > Regards,
> > > Ken
> > > 
> > > On Fri, Jul 22, 2016 at 02:27:01PM -0500, Kenneth Marshall wrote:
> > > > On Sat, Jul 16, 2016 at 12:04:26AM +1000, Bron Gondwana via Info-cyrus wrote:
> > > > > On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote:
> > > > > > On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> > > > > > > So if the issue apparently lies with twoskip, can we keep our dbs using 
> > > > > > > skiplist and do the 2.4 -> 2.5 upgrade?  Is it possible -h could revert 
> > > > > > > back to skiplist?
> > > > > > 
> > > > > > You can convert a db back just by changing the setting in imapd.conf and restarting - it will convert itself.
> > > > > > 
> > > > > > mboxlist_db: skiplist
> > > > > > 
> > > > > > > If it could help we could test upgrading to 2.5.8 on our dev server 
> > > > > > > while leaving our database(s) as skiplist.
> > > > > > 
> > > > > > Twoskip should be fixed in 2.5.9.  I've worked out what went wrong, and am working on patches now :)
> > > > > 
> > > > > Patches have passed all the tests - they're on master and 2.5.  I'm pushing to FastMail's testing stores and then going to bed :)
> > > > > 
> > > > > -- 
> > > > >   Bron Gondwana
> > > > >   brong at fastmail.fm
> > > > > ----
> > > > 
> > > > Hi Bron and Ellie,
> > > > 
> > > > Thank you for the new release. I am currently testing my upgrade process and while
> > > > I can access my mailboxes after the upgrade to 2.5.x but before the 'reconstruct -V max',
> > > > I cannot move messages between mailboxes. When I try I get the following error:
> > > > 
> > > > imap_copy_messages [a0012 NO Mailbox format corruption detected]?
> > > > 
> > > > and the move fails. Once I have updated the folder formats with the reconstruct,
> > > > it works, but this causes a lot of issues with the upgrade. Have you seen this
> > > > before and is there anything that can be done to make it work before the 
> > > > reconstruct?
> > > > 
> > > > Regards,
> > > > Ken
> > > *** HEADER EXTRACTED deferred/C/CB16F4219066 ***
> > > *** MESSAGE FILE END deferred/C/CB16F4219066 ***
> > 
> > 
> > -- 
> >   Bron Gondwana
> >   brong at fastmail.fm

> diff --git a/imap/mailbox.c b/imap/mailbox.c
> index aa933b9..98a6c35 100644
> --- a/imap/mailbox.c
> +++ b/imap/mailbox.c
> @@ -610,6 +610,10 @@ int mailbox_append_cache(struct mailbox *mailbox,
>  			  record->cache_offset, &record->crec);
>      if (r) return r;
>  
> +    /* old-style record */
> +    if (!record->cache_crc)
> +	return 0;
> +
>      if (record->cache_crc != crc32_buf(cache_buf(record)))
>  	return IMAP_MAILBOX_CHECKSUM;
>

More information about the Info-cyrus mailing list