blake2 and hash upgrades

[Bron Gondwana]

Date: November 23, 2016

That's when I started the draft for this and never got around to finishing it...

I'm seriously considering replacing sha1 in the index files with blake2 and upgrading all indexes to the new format!  This is a serious breaking change, and it's going to need an index format change.

Of course, the problem here is that there's no upgrade path from sha1 encoded into the sync protocol, so my first step is actually going to be adding an upgrade path, like so:

If the value is 20 bytes long (40 hex chars) it's a sha1.

If it's any longer, it's a http://multiformats.io/multihash/.  In particular, this means that all existing index will upgrade to store the following values in the larger space for hashes:

1114<sha1>

And the new cyrus.index file format will have a large enough space to store the new format.

Likewise all blobIds will be blake2 for newly delivered messages.  I will work on the theory that existing messages retain their sha1 value and sha1 blobIds, while new messages get new blake2 value and blake2 blobIds.

The 'G' key in the conversations DB will be a 20 byte value for sha1 without the additional prefix, while for other formats we're going to need to choose a different first character.

The big question is - do we drop everything and get this in BEFORE 3.0?  It means pushing the release date back at least a little bit, but since nobody (except FastMail) is using the conversations code and the 'G' keys yet (I hope), we can do it with only FastMail needing to go through the painful upgrade.  And then it's done!

(this will require replicas to be upgraded first, as our advice already says, because replicas will need to be able to receive multihash values as the GUID)

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm