*IMPORTANT* - bugfix sync_append_commit index breakage

[Bron Gondwana]

On Sun, Sep 02, 2007 at 01:33:11PM +1000, Rob Mueller wrote:
>
>>> I believe so.  If we also used it for appends, we wouldn't run into the
>>> problem that Fastmail is seeing.  But, it may be too much of a
>>> performance hit.
>
>> I don't think that any of the code does the .NEW for appends because that
>> makes append O(N) where N is the mailbox size, and that would suck for
>> huge mailboxes.
>
> Definitely. Rewriting the cyrus.index file for every message delivered to a 
> mailbox would be horrible, especially when some users have 100,000+ 
> messages in a mailbox.
>
>> Speaking of which, is there any reason why cyrus.expunge isn't sorted
>> by UID?  We have to rewrite the entire cyrus.index each time we do an
>> expunge anyway.
>
> I'm guessing it's because messages can be deleted + expunged in any order, 
> so it's easy just to append the record for any message being expunged to 
> the end of the cyrus.expunge file. If you had to keep it sorted, that would 
> mean every time you expunged a message, you'd have to reread, resort, and 
> rewrite the cyrus.expunge file. Generally the cyrus.expunge will be a lot 
> smaller than the cyrus.index, but still annoying...

Well, you are rereading (not sorting) the cyrus.index file every time
you do an expunge anyway.

A faster alternative would be to combine the two files again, have two
different counts (exists, n_records) - where exists may be smaller.
Also, you'd need to set aside a server flag for "really, really
expunged" in each index record.

There's the case of non-UID numbered fetch not working so well any more,
but honestly, who sane uses that in this concurrent world?

It's a way invasive patch though.

Bron.