Cyrus 2.5 status

[Bron Gondwana]

On Sat, Jan 10, 2015, at 12:51 PM, Gerd v. Egidy wrote:
> > > This will make finding mailboxes in filebased backups a lot harder,
> > > as we would have to restore the mailboxes.db from backup first to get
> > > the uniqueid. Using softlinks (mailboxname == link name, link points
> > > to the uniqid)
> > > might work depending how the backup program shows links.
> 
> I second this.

This is how I do it in our current FastMail backup format.

> > I would put the history of names that it had in the cyrus.header too, so it
> > can be recovered.
> 
> This will only help finding the correct name when you have a folder, but not 
> help finding a folder when you only got the path+name.

I don't understand what you mean.  From the cyrus.header file you can easily
see the folder name.  From the uniqueid in the mailboxes.db (new format), you
can figure the path name.  It will be something like `cyr_info datapath $foldername`
to convert that into a path.  Simpler than calculating it yourself based on the
fulldirhash such settings.

> > Harder to find individual mailboxes, sure.  I don't care though.  We should
> > be doing backups better, and good tooling will fix this.  The advantages of
> > super fast atomic deep renames outweigh the problems.
> 
> I do not agree. 
> 
> Having a fallback to files/directory level and being able to recreate a sane 
> mailbox state from there has saved me several times. In several occasions the 
> mailboxes.db was broken beyond repair (cyrus bugs in ver 2.3, machine lockups, 
> faulty ram without ecc,...), in other cases the filesystem was broken and only 
> parts of it were readable. The customer was lazy with backups so I had to work 
> with what was available.

It will be possible to create the recovery unless cyrus.header is corrupted AND
mailboxes.db is corrupted, in which case you would need to inspect the individual
emails to see who owned them.

But that's the case already if the filesystem is corrupted such that the folder path
is lost, as I have seen happen.  It gets to being a very unlikely case that you're
optimising for.  On the flip side, disk file path lengths are unpredictable with the
folder name encoded into the on-disk location, which has its own interesting
issues.  And lack of reliable fast rename has been an issue for a lot of people a
lot of the time.

> It is also a big plus for debugging & maintenance purposes to be able to 
> directly see the folders on the filesystem. E.g. doing a grep, lsof, ftop, 
> ncdu,...

Sure.  It's getting in the way of some serious optimisation possibilities though,
and Cyrus on-disk structures are really supposed to be pretty much opaque.

> So please keep a way to navigate the folder structures on disk. Using symlinks 
> would be the obvious choice and would introduce only marginal load when moving 
> folders.

I can see your point - but I still think that a cyr_info command which spits out the
right path gives you most of the navigation advantages, while not needing to add
tons of complex rollback code to rename.

Bron.


-- 
  Bron Gondwana
  brong at fastmail.fm