Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

[Bron Gondwana]

Thanks for reporting this.  Ellie, if you get a chance can you look at it?  I'm in the middle of the the FastMail security rollout right now, so I can't do anything today.

Bron.

On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote:
> Hi,
> 
> I accidentally hijacked another thread. So here is a new message. I am
> testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
> moving messages between folders before the 'reconstruct -V max' has been
> run on the mailboxes. Here is the error from the IMAP client:
> 
> imap_copy_messages [a0080 NO Mailbox format corruption detected]?
> 
> and the details from the IMAP logs:
> 
> <1469384341<a0077 NOOP
> >1469384341>a0077 OK Completed
> <1469384354<a0078 LIST "" "%"
> >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0078 OK Completed (0.240 secs 356 calls)
> <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> a0079 OK Completed
> <1469384361<a0080 UID COPY 1162729 "Drafts"
> >1469384362>a0080 NO Mailbox format corruption detected
> <1469384370<a0081 LIST "" "%"
> >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0081 OK Completed (0.230 secs 356 calls)
> <1469384372<a0082 CLOSE
> a0083 MYRIGHTS "Drafts"
> a0084 SELECT "Drafts"
> >1469384372>a0082 OK Completed
> >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> a0083 OK Completed
> >1469384372>* 0 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> * OK [UIDVALIDITY 1345619846] Ok
> * OK [UIDNEXT 6] Ok
> * OK [HIGHESTMODSEQ 1] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a0084 OK [READ-WRITE] Completed
> >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0078 OK Completed (0.240 secs 356 calls)
> <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> a0079 OK Completed
> <1469384361<a0080 UID COPY 1162729 "Drafts"
> >1469384362>a0080 NO Mailbox format corruption detected
> <1469384370<a0081 LIST "" "%"
> >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0081 OK Completed (0.230 secs 356 calls)
> <1469384372<a0082 CLOSE
> a0083 MYRIGHTS "Drafts"
> a0084 SELECT "Drafts"
> >1469384372>a0082 OK Completed
> >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> a0083 OK Completed
> >1469384372>* 0 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> * OK [UIDVALIDITY 1345619846] Ok
> * OK [UIDNEXT 6] Ok
> * OK [HIGHESTMODSEQ 1] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a0084 OK [READ-WRITE] Completed
> <1469384392<a0085 NOOP
> >1469384392>a0085 OK Completed
> <1469384412<a0086 NOOP
> >1469384412>a0086 OK Completed
> 
> Is this a known issue with the upgrade process? Messages can be delivered to
> the folders via a sieve script, but both message moves and copies fail. If there
> is any fix for this issue, it would be greatly appreciated. I have been waiting
> since the release of 2.4.x for a version that would handle the I/O storm caused
> by automatically upgrading the mailbox format on first access, and 2.5.x does
> manage that successfully. Thank you for any assistance.
> 
> Regards,
> Ken
> 
> On Fri, Jul 22, 2016 at 02:27:01PM -0500, Kenneth Marshall wrote:
> > On Sat, Jul 16, 2016 at 12:04:26AM +1000, Bron Gondwana via Info-cyrus wrote:
> > > On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote:
> > > > On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> > > > > So if the issue apparently lies with twoskip, can we keep our dbs using 
> > > > > skiplist and do the 2.4 -> 2.5 upgrade?  Is it possible -h could revert 
> > > > > back to skiplist?
> > > > 
> > > > You can convert a db back just by changing the setting in imapd.conf and restarting - it will convert itself.
> > > > 
> > > > mboxlist_db: skiplist
> > > > 
> > > > > If it could help we could test upgrading to 2.5.8 on our dev server 
> > > > > while leaving our database(s) as skiplist.
> > > > 
> > > > Twoskip should be fixed in 2.5.9.  I've worked out what went wrong, and am working on patches now :)
> > > 
> > > Patches have passed all the tests - they're on master and 2.5.  I'm pushing to FastMail's testing stores and then going to bed :)
> > > 
> > > -- 
> > >   Bron Gondwana
> > >   brong at fastmail.fm
> > > ----
> > 
> > Hi Bron and Ellie,
> > 
> > Thank you for the new release. I am currently testing my upgrade process and while
> > I can access my mailboxes after the upgrade to 2.5.x but before the 'reconstruct -V max',
> > I cannot move messages between mailboxes. When I try I get the following error:
> > 
> > imap_copy_messages [a0012 NO Mailbox format corruption detected]?
> > 
> > and the move fails. Once I have updated the folder formats with the reconstruct,
> > it works, but this causes a lot of issues with the upgrade. Have you seen this
> > before and is there anything that can be done to make it work before the 
> > reconstruct?
> > 
> > Regards,
> > Ken
> *** HEADER EXTRACTED deferred/C/CB16F4219066 ***
> *** MESSAGE FILE END deferred/C/CB16F4219066 ***


-- 
  Bron Gondwana
  brong at fastmail.fm