Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?

[ellie timoney]

Hi Binarus,

> Could anybody please shortly explain why? What exactly are the 
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been 
> quite sure that it just uses the IMAP protocol, but there seems to be 
> more to it ...

You're right, but missing a detail.  The XFER command is an IMAP extension, for use in cyrus clusters ("murders"), where mailboxes need to be identical between servers.  I don't know much about the details, but it makes sense to me that it would copy the mailboxes across at a "deeper" level, to make sure internal identifiers and such match.

cyradm uses the IMAP protocol – think of it as a command-driven IMAP client, except that it knows about cyrus extensions (like XFER) that generic IMAP clients (like imapsync) probably don't

Something like imapsync would probably be using the CREATE <mailbox> and APPEND <message> commands instead, which let the destination server choose its own internal identifiers, and which will probably be different from the source ones.  But they would be stored in the exact method the destination server prefers (such as using the newest version of the mailbox index format...)

> Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max).

That's not the "heaviest" form of reconstruct, it's just updating the mailbox indexes to the "max"imum (-V)ersion supported by the server.  This ends up adding a bunch of extra information to the indexes that the newer server version can make use of.  Generally if the index version is too old for what it's trying to do, it'll fall back to calculating missing values the long way, or provide less info, or just refuse the requested operation.  The assertion failure you saw is clearly a bug, but this smells familiar, so maybe it's already been fixed.  The latest version of the 2.5 release is 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth.

Hope this helps!

ellie

On Sun, Dec 16, 2018, at 9:06 PM, Binarus wrote:
> Dear all,
> 
> yesterday, I have moved a bunch of user mailboxes and public folders 
> from a server running 2.4.16 to another server running 2.5.10, using 
> cyradm's xfer command (see my previous messages).
> 
> After having finished the migration, I noticed a weird behavior in 
> Thunderbird (which is our standard email client): When trying to move a 
> message from the Inbox to the Junk or Trash folder, the message 
> disappeared from the Inbox for a short time, then reappeared. The log 
> files on the server were showing the dreaded entries:
> 
> Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: 
> assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(&record-
> >guid)
> 
> "Fortunately", a lot of other users are affected by such log entries and 
> weird email client behavior as well, so finding the solution on the 'net 
> was not too difficult: Running reconstruct did not lead to anywhere, but 
> running reconstruct -V max was the solution.
> 
> This lets me scratch my head:
> 
> In the past, I have upgraded Cyrus imapd at least three times, each time 
> using imapsync (instead of cyradm's xfer command) to move the mailboxes 
> and the public namespace (including all messages and subfolders) from 
> the old server to the new one. In none of these cases, it was necessary 
> to reconstruct anything afterwards. This would have been illogical 
> anyway: Each time, the new server had been setup from scratch, and no 
> mailboxes / messages / folders / subfolders had been moved directly via 
> file transfer from the old to the new server.
> 
> Now that I have used cyradm's xfer command to relocate the mailboxes / 
> messages / folders / subfolders, I surprisingly had to run the 
> "heaviest" form of reconstruct (-V max).
> 
> Could anybody please shortly explain why? What exactly are the 
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been 
> quite sure that it just uses the IMAP protocol, but there seems to be 
> more to it ...
> 
> Thank you very much for any insight,
> 
> Binarus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus