Future branch update - 28 April

[Bron Gondwana]

So I'm obviously not going to have anything ready
by the end of April.  *sigh*.

Basic status: non replicated IMAP is looking pretty
stable now.  NNTP is a disaster zone, I'm going to
have to spend some quality time on it later.
cyr_expire still needs some TLC.  LMTP and POP
should be fine, though I haven't been testing
them.

The rest of this email is a long ramble about my
thoughts on replication and where I am now.

tl;dr - pretty much rewriting replication to reduce
round trips and minimise bandwidth use - leveraging
the modseq numbers to know which records to update.

-----

I'm part way through gutting replication.  Throwing 
away a _lot_ of how it currently works, and replacing
with a more modseq based "set of changes" approach.

Basically the workflow for a set of mailboxes
(including a user) will be like this:

Client -> Server:
MAILBOXES user.a user.b.foo user.b.bar ...

Server -> Client:
* MAILBOX $unqid user.a $uidvalidity $lastuid $highestmodseq ...
* MAILBOX $unqid2 user.b.bar ...
OK MAILBOXES

At which point we can detect a rename from user.b.foo to
user.b.bar.  But I get ahead of myself.

We read the local mailboxes list, and from lastuid we can see
if there are any records that need appending.  We generate

RESERVE (user.a user.b.bar) $GUID1 $GUID2 $GUID3 ...

And get back

* RESERVE $GUID1 $GUID3 ...
OK RESERVE

So we know that GUID1 and GUID3 were reserved on the
other end, but GUID2 could not be reserved.  Note that
we provided the list of mailboxes to scan based on what
the server returned above.  All mentioned mailboxes get
scanned, so we will get the single-instance-store benefit.

Again - note the reduced round-trip IO.  The current
replication code calls out to each folder in turn for
reservations based on the list they return.  The new
code doesn't ever get the full list across the wire,
because that's too IO intensive.

Having reserved all the messages, the next step is to
update the mailboxes.  This is done in one command
per mailbox:

UPDATE $unqid user.a $partition $uidvalidity $acl
$highestmodseq $lastuid $recentuid ... \
STORE $uid1 $modseq1 ($flags1) \ STORE $uid2 $modseq2 \
($flags2) ... APPEND $uidN $internaldateN $modseqN \
($flagsN) $guidN {data} APPEND $uidM ...

Basically ALL the changes required to bring the mailbox
up to date in a single "transactional" change.  I am
seriously considering making this change be CONDSTORE
style dependent on the previous highestmodseq value not
having changed in the meantime!  This can be achieved
by a 2-stage process for the above, basically first a
"SELECT" style command which locks and checks, followed
by the actual contents.  It means an additional round
trip in the common case though, so maybe not.

Anyway, the "user" command is similar, except that it
obviously returns a few more details, and updates a few
more details.

I have changed from the "*", "**", "***" unsolicited
code magic.  I hate it.  Different lengths have different
meanings for different commands.  Instead they are ALL
'*' followed by a keyword, just like IMAP.  Means I can
write one generic parser for any command and have it
update the necessary lists.  Much less copy-pasta.

Oh yeah - LOCK/UNLOCK - totally gone.  All the sync
primatives use fully sane per-mailbox locking now, so
it should be possible to run multiple sync_client
sessions in parallel just fine.  SELECT is gone too.
The only command which affects multiple mailboxes at
once (with locking!) is RENAME.  The semantics on
RENAME should be fine because we've already had to
deal with lock inversion risks in rename for the
imapd - and we'll be using the same locking.  I've
added "partition" to the rename command, so renames
across partitions will replicate.

I do ramble so.  Code is getting pushed to github and
I'm being good and not rebasing it all together into
a mush at the moment, so you can see all the evolution
as I fiddle with things!  I'll squash the changes down
to sensible steps at some point.

Bron.