Replication and single instance store

[Bron Gondwana]

On Tue, 4 Sep 2007 10:09:40 +0100 (BST), "David Carter" <dpc22 at cam.ac.uk> said:
> On Tue, 4 Sep 2007, Bron Gondwana wrote:
> 
> > Ah - yeah, that's right.  Except that the restart only got negotiated 
> > after each folder was processed, and if you're pushing a new folder with 
> > 200,000 messages (say, after a user move in our case) then that got a 
> > bit memory hungry and all sorts insane.
> 
> Yes, this was the 5*MAX_MAILBOX_PATH allocation for each message when
> support for partitions was added.
> 
> My original code cached 300k messages on the server between restarts 
> (without any substantial memory leak). Better, but not perfect.
> 
> > Does this mean there is no way to get single-instance-store on a replica
> > if you're rebuilding it from scratch?
> 
> No. You would need a database which maintained a persistent mapping 
> between UUID and a list of files on each partitition which are that UUID.

I'm going to find out exactly how big this database is.  I'll be storing

KEY: UUID
VALUE: domain!user.path SPACE UID (TAB another!user.mailbox SPACE anotherUID)*

I don't think you can make it shorter than that without doing skanky things
like mailbox UniqueId, and then you'd want that stored in the mailboxes.db
(I'm not saying that's a bad thing!) or else two databases - another
containing a mapping from UniqueID to mailbox name (also, I see some value
here!)

Too many choices...

Ok, time to slap together a quick script to build this database on a current
store.  Should be pretty easy.  Unfortunately cyr_dbtool actually sucks for
this, so I'll be doing something about that first I guess.  So much for my
quiet night in!

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm