Is "reconstruct -V max..." needed when upgrading from 2.5.10 to 3.0.7?

[Savvas Karagiannidis]

Hi Nic,
as far as I know there are currently a couple of nasty open issues when
upgrading old mailboxes, created from earlier versions of cyrus, that have
not been upgraded to v13 index using reconstruct. These issues are:

https://github.com/cyrusimap/cyrus-imapd/issues/2171
https://github.com/cyrusimap/cyrus-imapd/issues/2208

The main problem is frequent crashes with error message "assertion failed"
when accessing these mailboxes.

In issue #2208 a reconstruct command was proposed as a workaround:
reconstruct -G -V max [mailboxname]

Note that -G parameter seems necessary, otherwise the problem persists...

Note also, that this should be done while still in version 2.5.x! The
behavior of version 3.x reconstruct is not the same and you will not have
the same results.

As for the cyrdump you mention, it would be better if you compared the
results before running reconstruct and after that.
I guess what you would see is that the uidlist before would probably be in
the range 6 - 9 (or you'd see a crash). cyrus 3.0.x considers that part of
the index corrupt so simply cannot access those uids (1-5). Reconstruct
detects this issue and what it does is that *reappends *those "newly found"
messages, giving them a new uid, so that's what uids 10-14 probably really
are. Note that this causes these messages to appear unread, since they have
no flags (not something you want for your clients' mailboxes).

Bron mentioned something about this bug in the cyrus devel list a few days
ago:
https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/thread.html#4313
but I didn't see any related commit or anything mentioned in the above
issues, so I'm not sure if there is a fix at the moment...

Hope this helps,
Regards,
Savvas Karagiannidis

On Tue, Jul 31, 2018 at 6:43 PM Nic Bernstein <nic at onlight.com> wrote:

> Friends,
> I'm preparing for a couple of belated 2.5.X to 3.0.X upgrades, and have a
> question about how necessary it is to run "reconstruct -V max" on the
> mailstore.  Both systems are currently running 2.5.10, and are already at
> index version 13.  However, when performing "reconstruct -V max" on one, on
> a new 3.0.7 (with patches) system, I see this:
>
> root at newmail:~# /usr/lib/cyrus/bin/reconstruct -V max user/onlight
> user.onlight uid 1 rediscovered - appending
> user.onlight uid 2 rediscovered - appending
> user.onlight uid 3 rediscovered - appending
> user.onlight uid 4 rediscovered - appending
> user.onlight uid 5 rediscovered - appending
> user/onlight
> Repacked user/onlight to version 13
>
> The last line can be ignored, as it's really a noop.  The "rediscovered -
> appending" stuff is what catches my eye.  However, once the reconstruct is
> complete, here's what the mailbox looks like:
>
> root at newmail:/var/spool/cyrus/mail/I/user/onlight# /usr/lib/cyrus/bin/cyrdump user/onlight
> Content-Type: multipart/related; boundary="dump-27466-1533049817-351841533"
>
> --dump-27466-1533049817-351841533
> Content-Type: text/xml
> IMAP-Dump-Version: 0
>
> <imapdump uniqueid="710a47ca47ebc676">
>   <mailbox-url>imap://newmail.example.com/user.onlight</mailbox-url>
>   <incremental-uid>0</incremental-uid>
>   <nextuid>15</nextuid>
> *  <uidlist>6 7 9 10 11 12 13 14 </uidlist>*
>
>   <flags>
> ...
>
> Note that the <uidlist> doesn't list those low number UIDs which were
> listed in the reconstruct sequence.  In other words, I think this all is
> harmless, but I'm not sure how much overhead it brings to the whole
> process.
>
> One of the servers has a total of 70GB of mail, so a complete reconstruct
> run only takes a short while.  The other, however, has over 8TB scattered
> over >30 partitions.  If I could avoid running reconstruct across that
> whole wad, it'd be great.
>
> Thoughts please?
>     -nic
>
> --
> Nic Bernstein                             nic at onlight.com
> Onlight, Inc.                             www.onlight.com
> 6525 W Bluemound Road, Suite 24           v. 414.272.4477
> Milwaukee, Wisconsin  53213-4073
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180731/e6a989fa/attachment.html>