Sophia database => fail

[Bron Gondwana]

On Sun, Jun 19, 2016, at 08:16, Howard Chu wrote:
> If by multi-level you mean LSM style DBs, you should be aware that these are 
> extremely fragile in the face of application and OS crashes. It's a 
> consequence of the fact that the DB state and metadata is scattered across 
> multiple files, and you cannot atomically update all of them at once. LevelDB 
> and RocksDB are both prone to self-corruption after a crash.

Great :(  *sigh*

> (Google "rocksdb corruption" ...)
> 
> Not quite sure I understand your use case. It sounds like you'll maintain a 
> local store, with recent records, and a remote store with older records. Is 
> that correct?

For the database I'd keep everything local, but I want to write everything back to
the store such that a host crash doesn't lose any data - and I don't want to amplify
writes out to absurdity doing it - though some level of amplification is obviously
OK.

> One of the features we'll be adding in LMDB 1.0 is incremental backup. This 
> will write out all the DB pages that are newer than a particular 
> transactionID. (A prototype of this feature is in the "incre" branch 
> https://github.com/LMDB/lmdb/tree/incre but it needs some rework before 
> releasing in 1.0)

That's definitely a step in the direction I need.

A good example of what I want is the way that the xapianactive file works in Cyrus
search at FastMail:

https://blog.fastmail.com/2014/12/01/email-search-system/

Because only the most recent database is writable (in this case on tmpfs, because
we don't need 100% reliability for search, it only takes about 20 minutes to scan
every mailbox and reindex the stuff that was on tmpfs after a crash)

Every other database is read-only - and you can compact multiple of them together
into a single database and then atomically switch the old ones out and the new one
in with a single very quick xapianactive rewrite - so it's acceptable to stop the world
while doing that.

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm