Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

[Kenneth Marshall]

Hi,

I accidentally hijacked another thread. So here is a new message. I am
testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
moving messages between folders before the 'reconstruct -V max' has been
run on the mailboxes. Here is the error from the IMAP client:

imap_copy_messages [a0080 NO Mailbox format corruption detected]?

and the details from the IMAP logs:

<1469384341<a0077 NOOP
>1469384341>a0077 OK Completed
<1469384354<a0078 LIST "" "%"
>1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" DSPAM_notspam
* LIST (\HasChildren) "/" DSPAM_spam
* LIST (\HasNoChildren) "/" Drafts
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren) "/" Sent
* LIST (\HasNoChildren) "/" Spam
* LIST (\HasNoChildren) "/" Trash
a0078 OK Completed (0.240 secs 356 calls)
<1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
>1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
a0079 OK Completed
<1469384361<a0080 UID COPY 1162729 "Drafts"
>1469384362>a0080 NO Mailbox format corruption detected
<1469384370<a0081 LIST "" "%"
>1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" DSPAM_notspam
* LIST (\HasChildren) "/" DSPAM_spam
* LIST (\HasNoChildren) "/" Drafts
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren) "/" Sent
* LIST (\HasNoChildren) "/" Spam
* LIST (\HasNoChildren) "/" Trash
a0081 OK Completed (0.230 secs 356 calls)
<1469384372<a0082 CLOSE
a0083 MYRIGHTS "Drafts"
a0084 SELECT "Drafts"
>1469384372>a0082 OK Completed
>1469384372>* MYRIGHTS Drafts lrswipkxtecda
a0083 OK Completed
>1469384372>* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
* OK [UIDVALIDITY 1345619846] Ok
* OK [UIDNEXT 6] Ok
* OK [HIGHESTMODSEQ 1] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a0084 OK [READ-WRITE] Completed
>1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" DSPAM_notspam
* LIST (\HasChildren) "/" DSPAM_spam
* LIST (\HasNoChildren) "/" Drafts
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren) "/" Sent
* LIST (\HasNoChildren) "/" Spam
* LIST (\HasNoChildren) "/" Trash
a0078 OK Completed (0.240 secs 356 calls)
<1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
>1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
a0079 OK Completed
<1469384361<a0080 UID COPY 1162729 "Drafts"
>1469384362>a0080 NO Mailbox format corruption detected
<1469384370<a0081 LIST "" "%"
>1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" DSPAM_notspam
* LIST (\HasChildren) "/" DSPAM_spam
* LIST (\HasNoChildren) "/" Drafts
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren) "/" Sent
* LIST (\HasNoChildren) "/" Spam
* LIST (\HasNoChildren) "/" Trash
a0081 OK Completed (0.230 secs 356 calls)
<1469384372<a0082 CLOSE
a0083 MYRIGHTS "Drafts"
a0084 SELECT "Drafts"
>1469384372>a0082 OK Completed
>1469384372>* MYRIGHTS Drafts lrswipkxtecda
a0083 OK Completed
>1469384372>* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
* OK [UIDVALIDITY 1345619846] Ok
* OK [UIDNEXT 6] Ok
* OK [HIGHESTMODSEQ 1] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a0084 OK [READ-WRITE] Completed
<1469384392<a0085 NOOP
>1469384392>a0085 OK Completed
<1469384412<a0086 NOOP
>1469384412>a0086 OK Completed

Is this a known issue with the upgrade process? Messages can be delivered to
the folders via a sieve script, but both message moves and copies fail. If there
is any fix for this issue, it would be greatly appreciated. I have been waiting
since the release of 2.4.x for a version that would handle the I/O storm caused
by automatically upgrading the mailbox format on first access, and 2.5.x does
manage that successfully. Thank you for any assistance.

Regards,
Ken

On Fri, Jul 22, 2016 at 02:27:01PM -0500, Kenneth Marshall wrote:
> On Sat, Jul 16, 2016 at 12:04:26AM +1000, Bron Gondwana via Info-cyrus wrote:
> > On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote:
> > > On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> > > > So if the issue apparently lies with twoskip, can we keep our dbs using 
> > > > skiplist and do the 2.4 -> 2.5 upgrade?  Is it possible -h could revert 
> > > > back to skiplist?
> > > 
> > > You can convert a db back just by changing the setting in imapd.conf and restarting - it will convert itself.
> > > 
> > > mboxlist_db: skiplist
> > > 
> > > > If it could help we could test upgrading to 2.5.8 on our dev server 
> > > > while leaving our database(s) as skiplist.
> > > 
> > > Twoskip should be fixed in 2.5.9.  I've worked out what went wrong, and am working on patches now :)
> > 
> > Patches have passed all the tests - they're on master and 2.5.  I'm pushing to FastMail's testing stores and then going to bed :)
> > 
> > -- 
> >   Bron Gondwana
> >   brong at fastmail.fm
> > ----
> 
> Hi Bron and Ellie,
> 
> Thank you for the new release. I am currently testing my upgrade process and while
> I can access my mailboxes after the upgrade to 2.5.x but before the 'reconstruct -V max',
> I cannot move messages between mailboxes. When I try I get the following error:
> 
> imap_copy_messages [a0012 NO Mailbox format corruption detected]?
> 
> and the move fails. Once I have updated the folder formats with the reconstruct,
> it works, but this causes a lot of issues with the upgrade. Have you seen this
> before and is there anything that can be done to make it work before the 
> reconstruct?
> 
> Regards,
> Ken
*** HEADER EXTRACTED deferred/C/CB16F4219066 ***
*** MESSAGE FILE END deferred/C/CB16F4219066 ***