Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?

[Binarus]

Dear ellie,

thanks again for answering my questions!

On 17.12.2018 23:38, ellie timoney wrote:
> Hi Binarus,
> 
>> Could anybody please shortly explain why? What exactly are the 
>> techniques and mechanisms cyradm's xfer uses to do its thing? I had been 
>> quite sure that it just uses the IMAP protocol, but there seems to be 
>> more to it ...
> 
> You're right, but missing a detail.  The XFER command is an IMAP extension, for use in cyrus clusters ("murders"), where mailboxes need to be identical between servers.  I don't know much about the details, but it makes sense to me that it would copy the mailboxes across at a "deeper" level, to make sure internal identifiers and such match.

OK, got it. I am glad that I at least didn't misunderstand cyradm
completely. I originally thought that xfer was just a command which
cyradm internally would translate into standard IMAP commands, which is
wrong as I have understood now.

> Something like imapsync would probably be using the CREATE <mailbox> and APPEND <message> commands instead, which let the destination server choose its own internal identifiers, and which will probably be different from the source ones.  But they would be stored in the exact method the destination server prefers (such as using the newest version of the mailbox index format...)

Then it actually was the right idea to try cyradm xfer instead of
imapsync. After all, we want our mailboxes, folders and messages
transferred to the new server without anything being altered as far as
possible, don't we? Now that I can use scripts with cyradm (thanks to
your answer to one of my other posts), this will be a pleasure next time.

>> Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max).
> 
> That's not the "heaviest" form of reconstruct, it's just updating the mailbox indexes to the "max"imum (-V)ersion supported by the server.  This ends up adding a bunch of extra information to the indexes that the newer server version can make use of.  Generally if the index version is too old for what it's trying to do, it'll fall back to calculating missing values the long way, or provide less info, or just refuse the requested operation.  The assertion failure you saw is clearly a bug, but this smells familiar, so maybe it's already been fixed.  The latest version of the 2.5 release is 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth.

OK, understood. In my case, it decided to refuse the operation ...
somehow. Obviously, it didn't tell the client that it refused, but
probably believed that the operation would be successful, sent an OK to
the client, began the operation (the message indeed disappeared from
Thunderbird's message list), noticed that it couldn't perform it and
then reverted it (the message re-appeared in Thunderbird's message list
about one second after it had disappeared).

This is no real issue to me even if the log entries are due to a bug, as
long as we can solve that problem by running the reconstruct. I just
wanted to know why we have to reconstruct; thanks again for the insight!

Regarding the version: In general, I don't have any problem with
compiling software myself. Unfortunately, I then am responsible for the
security updates as well, which I don't want, because I am not a full
time administrator (I have to care for apache2, ntp, cyrus, sendmail,
mysql and subversion servers, having only a few hours per week besides
my normal job for doing this). Therefore, I'd like the Linux
distribution / package manager to care for the security fixes, which
means that I have to use the distribution's stock packages.

However, this is an exceptional case because I will eventually have to
upgrade anyway due to the problem with the shared folder namespace
showing twice (described in one of my other posts). This is a thing
which our users (including myself) probably can't accept, so if
upgrading is the only remedy, I'll eventually do that. But I'll first
wait for some answers to that other post.

> Hope this helps!

It really did :-)

Thank you very much again!

Binarus