FastMail.FM patchset - new patches

[David Carter]

On Thu, 15 Mar 2007, Bron Gondwana wrote:

> * Make UUIDs work at all.  The initialisation order of the UUID
>  subsystem was wrong, so we had very few messages with a non-zero UUID.

message_uuid_master_init() should only be called after become_cyrus(). 
Every time that message_uuid_master_next_child() overflows, master needs 
to update MASTER_UUID_FILE.

Is MASTER_UUID_FILE owned by a user other than cyrus on your systems? I 
can't see any other obvious reason that moving message_uuid_master_init() 
up a few lines would help. service_create() is binding ports, which 
obviously has to happen as root, while masterconf_getsection() is just 
parsing master.conf. I am probably missing something obvious here, but the 
current ordering works for me.

> * MD5 UUIDs - we've created a new scheme for UUID generation, of
>  the format: 02[first 11 bytes of message file md5].  This allows
>  some basic integrity checking of the file on disk, and is still
>  plenty random.  Also adds the non standard IMAP FETCHable items
>  UUID, RFC822.MD5 (calculated on the fly), RFC822.FILESIZE (does
>  a stat or looks at the MMAP result if something else needs it)

I don't think that this is safe. It is important that the UUIDs really are 
unique, which is the reason for the paranoia in message_uuid_master_init.

The assertion:

   Is it safe? - we calulated that with one billion messages you have a one
   in 1 billion chance of a birthday collision (two random messages with
   the same UUID). They then have to get in the same MAILBOXES collection
   to sync_client to affect each other anyway.

Isn't the case: UUIDs span all MAILBOXES and APPEND event until a restart. 
If a UUID appears in one event and then is referenced by a second event 
some minutes later then the first message seen will be reused.

At the moment sync_client in 2.3 tracks the last few thousand messages by 
UUIDs. My original code tracked the last few hundred thousand messages 
(diminishing returns, but useful when seeding accounts). There is a much 
greater chance of collisions there than just comparing two messages.

-- 
David Carter                             Email: David.Carter at ucs.cam.ac.uk
University Computing Service,            Phone: (01223) 334502
New Museums Site, Pembroke Street,       Fax:   (01223) 334679
Cambridge UK. CB2 3QH.