Sophia database => fail

[Howard Chu]

Bron Gondwana wrote:
> On Sun, Jun 19, 2016, at 08:16, Howard Chu wrote:
>> Not quite sure I understand your use case. It sounds like you'll maintain a
>> local store, with recent records, and a remote store with older records. Is
>> that correct?
>
> For the database I'd keep everything local, but I want to write everything back to
> the store such that a host crash doesn't lose any data - and I don't want to amplify
> writes out to absurdity doing it - though some level of amplification is obviously
> OK.

> A good example of what I want is the way that the xapianactive file works in Cyrus
> search at FastMail:
>
> https://blog.fastmail.com/2014/12/01/email-search-system/
>
> Because only the most recent database is writable (in this case on tmpfs, because
> we don't need 100% reliability for search, it only takes about 20 minutes to scan
> every mailbox and reindex the stuff that was on tmpfs after a crash)
>
> Every other database is read-only - and you can compact multiple of them together
> into a single database and then atomically switch the old ones out and the new one
> in with a single very quick xapianactive rewrite - so it's acceptable to stop the world
> while doing that.

This sounds like a lot of bother, particularly the bit about "checking if 
tmpfs is full". It's also a bit confusing because you talk about "compacting" 
which I interpret as "cleaning out empty/unused space inside a DB" but in 
context it sounds like you really mean "merging" - combining multiple DBs into 
a single DB.

If I were building this system with LMDB there would be no separate temp and 
meta tiers. LMDB would just mmap the DB on the SSD and let the OS buffer cache 
keep the hot pages in RAM. I'm not really sure I'd bother with multiple DBs 
either, there's nothing to compact. The data tier would be no different from 
the meta tier.

When you say you can reindex the stuff on tmpfs quickly, that means you're 
only reindexing the most recent N emails?

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/