Sophia database => fail

Bron Gondwana brong at
Sat Jun 18 18:26:17 EDT 2016

On Sun, Jun 19, 2016, at 08:16, Howard Chu wrote:
> If by multi-level you mean LSM style DBs, you should be aware that these are 
> extremely fragile in the face of application and OS crashes. It's a 
> consequence of the fact that the DB state and metadata is scattered across 
> multiple files, and you cannot atomically update all of them at once. LevelDB 
> and RocksDB are both prone to self-corruption after a crash.

Great :(  *sigh*

> (Google "rocksdb corruption" ...)
> Not quite sure I understand your use case. It sounds like you'll maintain a 
> local store, with recent records, and a remote store with older records. Is 
> that correct?

For the database I'd keep everything local, but I want to write everything back to
the store such that a host crash doesn't lose any data - and I don't want to amplify
writes out to absurdity doing it - though some level of amplification is obviously

> One of the features we'll be adding in LMDB 1.0 is incremental backup. This 
> will write out all the DB pages that are newer than a particular 
> transactionID. (A prototype of this feature is in the "incre" branch 
> but it needs some rework before 
> releasing in 1.0)

That's definitely a step in the direction I need.

A good example of what I want is the way that the xapianactive file works in Cyrus
search at FastMail:

Because only the most recent database is writable (in this case on tmpfs, because
we don't need 100% reliability for search, it only takes about 20 minutes to scan
every mailbox and reindex the stuff that was on tmpfs after a crash)

Every other database is read-only - and you can compact multiple of them together
into a single database and then atomically switch the old ones out and the new one
in with a single very quick xapianactive rewrite - so it's acceptable to stop the world
while doing that.


  Bron Gondwana
  brong at

More information about the Cyrus-devel mailing list