ctl_cyrusdb questions, looking for answers

[Bron Gondwana]

On Tue, Mar 17, 2009 at 09:37:37AM -0400, Adam Tauno Williams wrote:
> > > Not exactly sure what you mean here. I think you only need the database in
> > > db.backup? and of course clean the environment before restoring such a db.
> > AFAIK only that three databases are copied to the db.backupX directory. So if 
> > there are other checkpointed dbs they stay in the original place and I need to 
> > backup them?
> 
> As I understand it: there is no reason to backup deliver.db,
> statuscache.db, or tls_sessions.db.  You can dump those without
> consequence and they will be rebuilt on the fly.

Yes - we delete these as part of our startup scripts (prefer not to
have any BDB files lying around after a restart)

Hey "checkpoint" is a massively overloaded term.  In skiplist,
a "checkpoint" is when the file gets rewritten in order.  It's
a complete rewrite of the file.  Otherwise, skiplist is append
only, so deleted or replaced records take up space.

The algorithm for checkpoints is currently:

tid->logend > (2 * db->logstart + SKIPLIST_MINREWRITE)

Which basically means checkpoint when the file is over half
garbage.  I think that's actually a little too rare, especially
with SKIPLIST_MINREWRITE = 16834, it means .seen files have a
lot of junk in them!

The "checkpoint" that ctl_cyrusdb does on the other hand, is
totally different.  It's just copying those files into the
backup directories.  It doesn't actually run a db checkpoint,
at least not on skiplist.  There's no way to trigger one
externally.

I have patches in the FastMail set that tune SKIPLIST_MINREWRITE
lower, and the amount of junk required lower.  They also do a
checkpoint every time we run recovery (recovery is basically
going through and recreating all the links between records, 
rolling back any uncompleted transaction as well).

I think the "checkpoint on recovery" code will go into 2.3.15 -
it makes sense to rewrite the file when it's hot.

Bron.