Cyrus Replication Rewrite

[Robert Mueller]

> a) add modseqs and tombstones everywhere

It's interesting how once this was in cyrus.index (tombstones = records
previously in cyrus.expunge), it was *clearly* right, and now that it's
not on any of the other meta data, how much it sucks trying to handle
deletes and/or updates on both sides. This will be a big win.

> Because of this speculative sending, it's never necessary to lock both
> ends for any change.  It's never necessary to be online.  A change
> either applies cleanly, or gets rejected.  Even "full mailbox sync"
> where you need to fetch all metadata from the other end to resolve a
> split brain will result in a final state which should apply cleanly to
> the other end.

This is interesting. If you reject a change because your state doesn't
match the start state, you then need to be able to ask the server for
"well I'm at state X, start from there" right?

So if you're applying a bunch of offline updates, you might get to the
end of that and be up to date, or you might get half way, have something
break, and then be out of sync until you can ask the master to start
from the right position again?

> e) with legality checking and ACL checking, this interface could be
>    exposed to regular users as well.  Obviously, they could only make
>    permitted changes to mailboxes they have permissions on - but along
>    with 'b' where the "master" actually has no knowledge of the
>    replica, a remote machine could connect and poll for changes.
...
> 1) move sync_server into imapd as a separate command: CYR_SYNC which
>    reads and replies with DLISTs.

Ah you're long term dream of "one cyrus in the cloud for reliability,
one cyrus on a local server for low latency" dream finally being able to
come true since you can just access the sync protocol over an IMAP
connection :)

> 2) write a 'backup' command line tool as a client to imapd which
>    connects and requests the bits it needs.
> 3) change sync_client to cache remote state for speculative initial
>    commands (no initial query required before shipping changes)

I like that we get to kill multiple birds with one stone. And we could
probably make more of our backup system public as well (at least the
client bit that reads diffs and builds .tgzs, and part of the restore
code), rather than using the custom daemon that peaks into cyrus
internals...

> For efficiency, we also need to re-add sync_crc caching.  This can
> either be done in something like statuscache, or put back in the index
> header.  I do like the idea of statuscache actually, because it

Good point, statuscache! I kept thinking cyrus.header, but statuscache
seems better since the canonical data is in cyrus.index, and the value
in statuscache can always be recalculated if statuscache is blown away,
which is what a "cache" is supposed to be about.

Rob