Cyrus Future status update

[Ken Murchison]

Bron,

What is broken in nntpd?  I don't know who is using it, but I will 
certainly fix it.


Bron Gondwana wrote:
> So - I haven't pushed anything more to CVS for a bit!  That's because I
> wanted to refactor some more list logic out of sync_support.c and into its
> own function where it can be used by mbdump and friends as well.
> 
> 1) Refactoring and Sync Protocol
> 2) QRESYNC and ImapTest
> 3) Still TODO
> 
> ------------------------
> 
> 1) Refactoring and Sync Protocol
> 
> I've created dlist.c and dlist.h - which is a "dumplist" format.  The spec
> looks something like this:
> 
> * Atoms just like IMAP
> * Literals just like IMAP
> * Lists just like IMAP - starts with '(', ends with ')'
> * New type: KEY-VALUE list - starts with '%(' ends with ')'.
> * New type: message-file literal: %{partition GUID $n}\r\n\<$n bytes>
> 
> The top level of a command is always a KEY-VALUE pair, with the value
> often being another KEY-VALUE list.  So a brand new user with a single
> test message in their mailbox might cause sync traffic like this:
> 
> NOTE: all sync traffic that's not support stuff (login, compress, etc) is
> done via two commands. "GET" which queries the server, and "APPLY" which
> makes changes.  Also, the opposites are "UN<command>", so RESET became
> UNUSER.
> 
> APPLY MESSAGE reserves the message in that partition on the server - so all
> update traffic includes the full record, and copies the file into place if
> needed.
> 
> Everything is named - there's no positional magic.
> 
> And of course there's SYNC_CRC.  I'll document that somewhere too :)
> 
> <1278460344<GET USER foo
>> 1278460344>NO IMAP_MAILBOX_NONEXISTENT No Such Mailbox
> <1278460344<APPLY UNUSER foo
>> 1278460344>OK success
> <1278460344<APPLY MESSAGE (%{default 196922b6d822b618c665874fb523b9058a0adb56 92}
> From: test <test at example.com>
> To: test <test at example.com>
> 
> Some stuff in the body...
> .
> )
>> 1278460344>OK success
> <1278460344<APPLY MAILBOX %(UNIQUEID 74b8ef164c33c1b8 MBOXNAME user.foo LAST_UID 1 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1278460344 POP3_LAST_LOGIN 0 UIDVALIDITY 1278460344 PARTITION default ACL "foo      lrswipkxtecda   admin   lrswipkxtecd    " TYPE NORMAL OPTIONS P SYNC_CRC 123100176 RECORD (
>   %(UID 1 MODSEQ 1 LAST_UPDATED 1278460344 FLAGS (\Flagged) INTERNALDATE 1268029091 SIZE 92 GUID 196922b6d822b618c665874fb523b9058a0adb56)))
>> 1278460345>OK success
> 
> ------------------------
> 
> 2) QRESYNC and ImapTest
> 
> I've been working with ImapTest from the Dovecot project to check that the
> index.c model is correct.
> 
> http://www.imapwiki.org/ImapTest
> 
> Since last night, I finally ironed out the last reporting bug (I hope) and
> ImapTest says that it's working correctly, both with and without QRESYNC.
> I've even got support for match_seq and match_uid to index_vanished! (though
> I don't think ImapTest uses them...)
> 
> ------------------------
> 
> 3) Still TODO
> 
> * A working EXPUNGE_IMMEDIATE codepath.
> * mbdump and restore with the dlist protocol
> * nntpd - I don't think it's actually working at all!  I'd love to hear from
>   someone who uses it and is willing to help test it.
> * tidying up the number of duplicate utilities.
> * a seen_cleanup script to remove the seendb entries for mailboxes which
>   have been upgraded to the user-internal-seen format.  I'm not making it
>   part of the inital upgrade because that would mean you lose seen data if
>   you decide to undo the upgrade and reconstruct with an earlier cyrus!
> * reconstruct - create a non-invasive mode, and think more about how to
>   handle reconstruct without exclusive locking, and reconstruct that will
>   be replication safe.  Lots to think about here...
> 
> ------------------------
> 
> Still, this is up and running on one testing store at FastMail now - including
> one of my users.  So my email is working on top of it :)  The locking system
> is really very good now - I'm proud of it.  Actual guaranteed consistency with
> the view on clients even under heavy load.
> 
> I'll keep pushing stuff upstream as I'm happy that it's fairly stable.  As
> always, you can see my latest code at http://github.com/brong/cyrus-imapd/
> 
> Regards,
> 
> Bron.
> 

-- 
Kenneth Murchison
Systems Programmer
Carnegie Mellon University