"reconstruct -k" is still broken: do not use.

[Bron Gondwana]

On Sun, Dec 23, 2007 at 04:53:48PM +0000, David Carter wrote:
>
> Guess who thought that the quietest week of the year was an excellent 
> chance to upgrade from 2.3.8 to 2.3.11 and "reconstruct -rkG" everything in 
> sight to get shiny new SHA1 GUIDs? Fortunately I am sufficiently paranoid 
> about my Cyrus machines that only a test system ended up with:
>
>  Dec 21 22:57:07 cyrus-1 cyr_expire[23922]:
>   IOERROR: reading cache record for user.dpc22:
>   got bogus offset 577922368 for 1/2; try reconstruct
>
> First up, a trivial xrealloc bug:
>
> reconstruct_expunge() tracks no more than 300 expunged messages in a 
> mailbox. Anything more than this becomes uninitialised space in a 
> xrealloc()ed array: typically zeros on my platform. Consequently messages 
> end up in both cyrus.index and cyrus.expunge when you reconstruct -k.
>
> http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3cvs/reconstruct3.patch

Ouch, yeah.  Grabbing that.  Thanks.

> Rather more fundamental: "reconstruct -k" copies expunge.index records 
> verbatim, without making any effort to update offsets into cyrus.cache. 
> These offsets are used by both "cyr_expire -X" and "unexpunge".

unexpunge I don't care about, because I think it's fundamentally broken
in an environment where you care about bandwidth anyway (bumping
UIDVALIDITY tends to annoy client implementations, who then go and
resync everything from scratch).

cyr_expire I worked around it here:

http://cyrus.brong.fastmail.fm/patches/cyrus-expunged-nocache-2.3.10.diff

Basically, who cares about the cache record for a deleted message.  It's
not like you can't reconstruct it from the rfc822 file on disk when you
recover it (if ever).  I'm much more concerned about losing \Seen
information for deleted messages, but I haven't come up with a good fix
for that short of major rewrites.

> cyrus.cache entries aren't constant: the cached headers and bodystructure 
> information changes from version to version. Here is a before and after 
> version just reconstructing a single cache file from 2.3.8 to 2.3.11
>
>   -rw-r----- 1 cyrus cyrus 622680 2007-12-23 15:56 cyrus.cache-
>   -rw-r----- 1 cyrus cyrus 626404 2007-12-23 15:56 cyrus.cache
>
> I know how I plan to fix this problem: reconstruct needs to rebuild 
> cyrus.index and expunge in parallel, rather like sync_combine_commit(). 
> However this will be a rewrite of reconstruct.c, not a one line fix.
>
> For the time being here is a small utility to clear GUIDs without the 
> overhead (and hopefully without the brokenness) of "reconstruct -rgk":
>
> http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3cvs/reset_guids.patch

I did this in Perl with my very own little pack and unpack routines that
are also capable of upgrading or downgrading index files as they stream
them, as well as plenty of other funkyness.  I'm sure I've posted the
library here before - though I should probably put it up somewhere on the
web for general consumption.  It also only supports v9/v10/(future v11
with crc32 fields).

Bron.