uniqueid based paths (was: Minutes 16/3)

[Thomas Jarosch]

Hi Bron,

On Tuesday, 17. March 2015 09:50:34 Bron Gondwana wrote:
> Bron:
>  - working on uniqueid-based-paths
>  - we'll need upgrade path for existing installs (maybe detect old path
> and rename-on-open?) - reconstruct et al will need support for the new
> paths.  Rebuild mailboxes.db from the new cyrus.header with all
> mailboxes.db data in it.

I'm still wondering if there's an easier solution to the uniqueid based
path problem while keeping a human readable filesystem structure.
That would make backup and restore with traditional unix backup tools
much easier. I'll post a small tool we developed for cyrus 2.4 later on.

Having fast, atomic renames is certainly a welcoming improvement,
the rename stuff has bitten us often, too.

My colleague Gerd v. Egidy wrote this the last time on the topic:
-------------------
Both are binary structures which are cyrus version dependent and require a 
working cyrus installation of this version to be of any value. This means 
selecting the files to restore from backup has to be tightly integrated with 
a running cyrus which really complicates things. 

It is also a big burden for version-independent long-term storage. Currently 
I can take a snapshot from a 10 year old cyrus version, throw away all 
mailboxes.db and cyrus.headers, and recreate a working cyrus 2.4 
installation  from that just using simple shell scripts. I lose flags, acls 
and other metadata, but I get all the mails and folders, and that is what 
counts most for the users.
-------------------


If we switch to uniqueid based paths, what about providing symlinks to a 
"virtual" directory structure? Updating symlinks would probably not be 
atomic since we can't do a cyrus.header update and update a symlink at the 
same time. Though we could only complete the "transaction" in mailboxes.db
when all steps are done. So this could work even when imapd
is shut down in the middle of a rename.

May be the symlinks might even help during the migration / upgrade phase?

What about writing out the folder information from cyrus.header
also as JSON text files? This file is not interpreted by cyrus normally,
but it could be used by other tools and aid long term storage of backups.

In our backup system, we only keep text dumps of mailboxes.db
and annotations.db for maximum compatibility.
Those get converted back on restore.

Cheers,
Thomas