Cyrus Replication Rewrite

[Bron Gondwana]

Yes, again ;)  Having written a rewrite once, I want to do it again.

First - WHY???

a) make master/master replication safe
b) make "pull" replication available
c) make replication and backup identical
d) make offline replication (diff shipping)
e) make replication available to regular users

OK, how?

a) add modseqs and tombstones everywhere

* "meta" database containing:
  1) contents of user.sub
  2) contents of user.seen
  3) names and sha1 (guid)s of all sieve files
  4) per-user modseq and uidvalidity stuff

Complete with modseqs and expiry of records on everything.  The mailboxes.db
and annotations.db will have the same things too - records get a modseq and
a "isdeleted" flag. 

Let's consider "per user" for a bit.  Where there's currently a "quotaroot",
there's no equivalent "userroot".  Users are special in various ways, but
this needs to be made more clear.  Non-user folders (shared hierarchies)
will need to be handled too.  They may even need nesting.

b,c,d) the main thing here is using (a) to ship just changes.  Replication of
       any piece of data will always include:
  1) a start point - either modseq 0, blank CRC or a particular modseq,
     CRC algorithm and CRC.  Given that data, you either know immediately
     that the destination is unchanged-since, or that you need to return an
     error and your own "changed since that point" for the remote end to
     integrate before continuing.

Because of this speculative sending, it's never necessary to lock both ends
for any change.  It's never necessary to be online.  A change either applies
cleanly, or gets rejected.  Even "full mailbox sync" where you need to fetch
all metadata from the other end to resolve a split brain will result in a
final state which should apply cleanly to the other end.

Speaking of which - changes are always "legal" - any attempt to reduce a
modseq, change a uidvalidity, unexpunge a UID - these are all rejected.
The uidvalidity is kind of a special case - it's a delete and recreate of
a mailbox, and is only legal with the UIDVALIDITY increasing.

Which leads to:

e) with legality checking and ACL checking, this interface could be exposed
   to regular users as well.  Obviously, they could only make permitted
   changes to mailboxes they have permissions on - but along with 'b' where
   the "master" actually has no knowledge of the replica, a remote machine
   could connect and poll for changes.

At which point you could add:

f) IDLE style notify on a replication connection, so you can connect, sync,
   and ask to monitor a set of mailboxes for changes and be notified
   (as sync change events) when they happen.

===================

Ok, that's a nice broad set of goals, how do we get there?

1) move sync_server into imapd as a separate command: CYR_SYNC which reads and
   replies with DLISTs.
2) write a 'backup' command line tool as a client to imapd which connects and
   requests the bits it needs.
3) change sync_client to cache remote state for speculative initial commands
   (no initial query required before shipping changes)

For efficiency, we also need to re-add sync_crc caching.  This can either be
done in something like statuscache, or put back in the index header.  I do
like the idea of statuscache actually, because it reduces the IO hit of sync
even further - and the CPU hit too.  Some of our servers are currently
overloaded on CPU due to the continual CRC recalculation.

One complexity with sync_crc is that annotations don't currently calculate
sync_crc changes as they are written.  That will need to be fixed before we
can really cache them.

The rest happens later... permissions and such :)

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm