FastMail.FM patchset - new patches

[Bron Gondwana]

All the patches mentioned here are available for download at:

http://cyrus.brong.fastmail.fm/


We've been busy working on a bunch of cyrus issues since I last
posted.  In particular things we have hit cleaning up from the
"sync_server left files lying around after a bail out that caused
random messages to be overwritten a lot later" bug.


Ken - some of these are definite candidates for upstream.  As much
as possible we've made new behaviour optional with the default
being the current behaviour.

REMAINING ON MY TODO LIST:
==========================
* undoable "DELETE"
* investigate why data path directories aren't always being created
  for domainsplit users but meta paths are on the replica.
* use the MD5 based UUID and the RFC822.MD5 to do spot-checks in our
  checkreplication script, similarly RFC822.SIZE and RFC822.FILESIZE.
* expand logging for message delivery to include the UUID and UID of
  the delivered message.


WHAT'S NEW:
===========

* Make UUIDs work at all.  The initialisation order of the UUID
  subsystem was wrong, so we had very few messages with a non-zero
  UUID.  Also, the documentation for provide_uuid was misleading/
  difficult to understand.  Make that clearer.

* MD5 UUIDs - we've created a new scheme for UUID generation, of
  the format: 02[first 11 bytes of message file md5].  This allows
  some basic integrity checking of the file on disk, and is still
  plenty random.  Also adds the non standard IMAP FETCHable items
  UUID, RFC822.MD5 (calculated on the fly), RFC822.FILESIZE (does
  a stat or looks at the MMAP result if something else needs it)

* Abort sync on mismatched UUIDs (to go with above) - make failover
  much safer to avoid deleting files that got delivered while the
  other machine was the replica.

* windows-1250/1251/1255 codepages - some users we having trouble
  searching for things in messages with these codepages.  I believe
  this is already accepted upstream.

* pcreposix - use the PCRE engine for regexp matching in sieve.  The
  gnu posix library had some nasty edge case behaviour which we're
  hoping won't exist in the prce library.

* sync_client permissions check - I thought this had gone in upstream
  but must have been missed.  Ken, please apply this one, it's a really
  obvious typo fix!

* Set internal date based on first Received header - this one is
  probably a lot more controversial - but we have a pile of messages
  all with an internaldate of November 2005 when we restored from
  backups and lost all internaldate information.  We also have a pile
  with bogus values thanks to having had mtimes way out due to being
  on a replica.

* Speaking of which, there's a patch to set the mtime of the file on
  the replica based on the internaldate as well!  We try to keep all
  these values in sync so re-creating the internaldate correctly is
  much easier.

* RENAME INBOX.  It's a cow no matter how you look at it, broken in one
  way or another, but the thing that hurt us was that it broke replication.
  The patch we chose makes replication work again at the expense of losing
  all seen state for the copied messages.  The alternative was much more
  invasive, because sync_server does the "rename" operation as the admin
  user and they don't get the special treatment.  Man I hate special cases.
  Moo.

* Statuscache.  There has been a pretty major rewrite.  It's a lot faster
  for the common case now, because it avoids statting the seen file, opting
  instead to make all codepaths that affect the seen file also delete the
  associated statuscache record.  Faster reads, slightly slower writes for
  the seen DB.  Guess what our users do more of.


Regards,

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm