uniqueid based paths (was: Minutes 16/3)

[Bron Gondwana]

On Fri, Mar 20, 2015, at 10:08 PM, Thomas Jarosch wrote:
> Hi Bron,
> 
> On Tuesday, 17. March 2015 09:50:34 Bron Gondwana wrote:
> > Bron:
> >  - working on uniqueid-based-paths
> >  - we'll need upgrade path for existing installs (maybe detect old path
> > and rename-on-open?) - reconstruct et al will need support for the new
> > paths.  Rebuild mailboxes.db from the new cyrus.header with all
> > mailboxes.db data in it.
> 
> I'm still wondering if there's an easier solution to the uniqueid based
> path problem while keeping a human readable filesystem structure.
> That would make backup and restore with traditional unix backup tools
> much easier. I'll post a small tool we developed for cyrus 2.4 later on.

Pretty sure we've already had this dance.  I did build a half-way-house
solution a few years ago, which looked like this:

b/user/brong/user.brong.subfolder/

It had the advantage that no email folder contained another folder, so you
could rename the folder on disk with a rename(2) call.

But it was still racy, it had filesystem length constraints, and is still more
expensive than the uniqueid-based rename.

But we haven't forgotten about backup and restore.

> My colleague Gerd v. Egidy wrote this the last time on the topic:
> -------------------
> Both are binary structures which are cyrus version dependent and require a 
> working cyrus installation of this version to be of any value. This means 
> selecting the files to restore from backup has to be tightly integrated with 
> a running cyrus which really complicates things. 

We are now going to be storing the ENTIRE mailboxes.db value in cyrus.header,
to the point that mailboxes.db can be lost, and you can just run a reconstruct
command to read the filesystem and rebuild it from cyrus.header.  Yay.

Indeed, we will not only have the current folder name, we'll also have the historical
list of folder names this folder ever had, along with CTIME and MTIME for the
mailboxes.db record on each one (MTIME of the deleted ones obviously being
deletion time) and the UIDVALIDITY they had.

If the folder is deleted, the top record will obviously be a DELETED.* foldername.

> It is also a big burden for version-independent long-term storage. Currently 
> I can take a snapshot from a 10 year old cyrus version, throw away all 
> mailboxes.db and cyrus.headers, and recreate a working cyrus 2.4 
> installation  from that just using simple shell scripts. I lose flags, acls 
> and other metadata, but I get all the mails and folders, and that is what 
> counts most for the users.

Yeah, or you could use the complete, incremental backup format that Ellie is
working on, based on my discussions with David Carter who wrote the original
replication system and various mailing list threads, plus the backup system which
works at FastMail already.

Then you don't lose flags, acls and other metadata either.

> If we switch to uniqueid based paths, what about providing symlinks to a 
> "virtual" directory structure? Updating symlinks would probably not be 
> atomic since we can't do a cyrus.header update and update a symlink at the 
> same time. Though we could only complete the "transaction" in mailboxes.db
> when all steps are done. So this could work even when imapd
> is shut down in the middle of a rename.

We already had this discussion too.  This happens really rarely.  A better solution
is a human readable and machine parseable version of the folder name in the
cyrus.header.

How often is your cyrus instance totally down?  Not very.  And it will be easy to
have a tool which can parse the mailboxes.db or cyrus.headers to find the
mailbox you need.

> What about writing out the folder information from cyrus.header
> also as JSON text files? This file is not interpreted by cyrus normally,
> but it could be used by other tools and aid long term storage of backups.

So JSON vs DLIST is an interesting question that I could be swayed on.
DLIST is more IMAPpy, but that's not necessarily the most important thing.

> In our backup system, we only keep text dumps of mailboxes.db
> and annotations.db for maximum compatibility.
> Those get converted back on restore.

That makes sense.

Now annotations.db is somewhat interesting.  I'm really tempted to store all
the mailbox annotations in a different file within the mailbox, potentially "as well".
The advantage of a central annotations.db is speed of looking up an annotation
on every single mailbox - but that's what statuscache and friends are for.
You find yourself paying the mailbox open price if you want to look at virtual
annotations like 'dupdeliver' anyway.

Then both mailboxes.d AND annotations.db would be rebuildable from just a
spool copy.

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm