Cyrus Future status update

[Bron Gondwana]

So - I haven't pushed anything more to CVS for a bit!  That's because I
wanted to refactor some more list logic out of sync_support.c and into its
own function where it can be used by mbdump and friends as well.

1) Refactoring and Sync Protocol
2) QRESYNC and ImapTest
3) Still TODO

------------------------

1) Refactoring and Sync Protocol

I've created dlist.c and dlist.h - which is a "dumplist" format.  The spec
looks something like this:

* Atoms just like IMAP
* Literals just like IMAP
* Lists just like IMAP - starts with '(', ends with ')'
* New type: KEY-VALUE list - starts with '%(' ends with ')'.
* New type: message-file literal: %{partition GUID $n}\r\n\<$n bytes>

The top level of a command is always a KEY-VALUE pair, with the value
often being another KEY-VALUE list.  So a brand new user with a single
test message in their mailbox might cause sync traffic like this:

NOTE: all sync traffic that's not support stuff (login, compress, etc) is
done via two commands. "GET" which queries the server, and "APPLY" which
makes changes.  Also, the opposites are "UN<command>", so RESET became
UNUSER.

APPLY MESSAGE reserves the message in that partition on the server - so all
update traffic includes the full record, and copies the file into place if
needed.

Everything is named - there's no positional magic.

And of course there's SYNC_CRC.  I'll document that somewhere too :)

<1278460344<GET USER foo
>1278460344>NO IMAP_MAILBOX_NONEXISTENT No Such Mailbox
<1278460344<APPLY UNUSER foo
>1278460344>OK success
<1278460344<APPLY MESSAGE (%{default 196922b6d822b618c665874fb523b9058a0adb56 92}
From: test <test at example.com>
To: test <test at example.com>

Some stuff in the body...
.
)
>1278460344>OK success
<1278460344<APPLY MAILBOX %(UNIQUEID 74b8ef164c33c1b8 MBOXNAME user.foo LAST_UID 1 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1278460344 POP3_LAST_LOGIN 0 UIDVALIDITY 1278460344 PARTITION default ACL "foo      lrswipkxtecda   admin   lrswipkxtecd    " TYPE NORMAL OPTIONS P SYNC_CRC 123100176 RECORD (
  %(UID 1 MODSEQ 1 LAST_UPDATED 1278460344 FLAGS (\Flagged) INTERNALDATE 1268029091 SIZE 92 GUID 196922b6d822b618c665874fb523b9058a0adb56)))
>1278460345>OK success

------------------------

2) QRESYNC and ImapTest

I've been working with ImapTest from the Dovecot project to check that the
index.c model is correct.

http://www.imapwiki.org/ImapTest

Since last night, I finally ironed out the last reporting bug (I hope) and
ImapTest says that it's working correctly, both with and without QRESYNC.
I've even got support for match_seq and match_uid to index_vanished! (though
I don't think ImapTest uses them...)

------------------------

3) Still TODO

* A working EXPUNGE_IMMEDIATE codepath.
* mbdump and restore with the dlist protocol
* nntpd - I don't think it's actually working at all!  I'd love to hear from
  someone who uses it and is willing to help test it.
* tidying up the number of duplicate utilities.
* a seen_cleanup script to remove the seendb entries for mailboxes which
  have been upgraded to the user-internal-seen format.  I'm not making it
  part of the inital upgrade because that would mean you lose seen data if
  you decide to undo the upgrade and reconstruct with an earlier cyrus!
* reconstruct - create a non-invasive mode, and think more about how to
  handle reconstruct without exclusive locking, and reconstruct that will
  be replication safe.  Lots to think about here...

------------------------

Still, this is up and running on one testing store at FastMail now - including
one of my users.  So my email is working on top of it :)  The locking system
is really very good now - I'm proud of it.  Actual guaranteed consistency with
the view on clients even under heavy load.

I'll keep pushing stuff upstream as I'm happy that it's fairly stable.  As
always, you can see my latest code at http://github.com/brong/cyrus-imapd/

Regards,

Bron.