CONDSTORE and replication

[Bron Gondwana]

We've enabled CONDSTORE on an entire "Store" here at FastMail
to get an idea of what's involved in supporting it.  (it's the
store that the FastMail staff are on, so we're testing it on
ourselves first!)

So - there's a problem where \Seen changes are not causing the
modseq information to be replicated.  I think the following
patch fixes the issue (need to run sync_client on all users
who are currently not in sync, then leave things running for
a bit to make sure), but there is a downside.

Each time you set a \Seen flag (including just viewing a
message without using .PEEK), it will cause the entire
mailbox metadata to be:

a) read from disk
b) copied across the wire

That kind of sucks, even more than just the IO cost of
writing a couple of bit64s to the index file (modseq
in the record and highestmodseq in the header).  David,
I've CC'd you on this to get your opinion on more
fine-grained replication logging for metadata operations.
sync_log_mailbox is a pretty heavy-weight flyswat for
single record changes.

I'm thinking something like

sync_log_index(mailbox->name, uidrange), where uidrange
contains a compressed range of UIDs (seen-db format)

The sync_client would then just send a the correct values
for:

a) all header values (including highestmodseq, lastmodified,
   uidnext, etc)

b) all values within just the listed index records, which
   would overwrite the records for the same UID in the
   target mailbox, unconditionally.

   (b1) modulo our "if the GUIDs don't match then log the
        issue and don't update the record at all" logic here
        at FastMail to avoid destroying delivered messages
        after a hard-failover, of course....


What do you think?  Sane?  I suspect it's a bit of work, but
it would reduce traffic a fair bit.  Probably wouldn't help
with IO though, since every index file algorithm seems to be
linear search based.

Bron.

diff --git a/imap/imapd.c b/imap/imapd.c
index acddf31..365df80 100644
--- a/imap/imapd.c
+++ b/imap/imapd.c
@@ -4564,7 +4564,7 @@ void cmd_store(char *tag, char *sequence, int usinguid)
 
 	/* We only need to log a MAILBOX event if we've changed
 	   a flag other than \Seen */
-	if (storeargs.system_flags || nflags ||
+	if (storeargs.system_flags || nflags || (imapd_mailbox->options & OPT_IMAP_CONDSTORE) ||
 	    storeargs.operation == STORE_REPLACE) {
 	    sync_log_mailbox(imapd_mailbox->name);
 	}