Sophia database => fail

[Howard Chu]

Howard Chu via Cyrus-devel wrote:
> Bron Gondwana wrote:
>> A good example of what I want is the way that the xapianactive file works in
>> Cyrus
>> search at FastMail:
>>
>> https://blog.fastmail.com/2014/12/01/email-search-system/
>>
>> Because only the most recent database is writable (in this case on tmpfs,
>> because
>> we don't need 100% reliability for search, it only takes about 20 minutes to
>> scan
>> every mailbox and reindex the stuff that was on tmpfs after a crash)

Also, since you're using tmpfs, this in-memory benchmark is relevant.
http://lmdb.tech/bench/inmem/
>>
>> Every other database is read-only - and you can compact multiple of them
>> together
>> into a single database and then atomically switch the old ones out and the
>> new one
>> in with a single very quick xapianactive rewrite - so it's acceptable to
>> stop the world
>> while doing that.
>
> This sounds like a lot of bother, particularly the bit about "checking if
> tmpfs is full". It's also a bit confusing because you talk about "compacting"
> which I interpret as "cleaning out empty/unused space inside a DB" but in
> context it sounds like you really mean "merging" - combining multiple DBs into
> a single DB.
>
> If I were building this system with LMDB there would be no separate temp and
> meta tiers. LMDB would just mmap the DB on the SSD and let the OS buffer cache
> keep the hot pages in RAM. I'm not really sure I'd bother with multiple DBs
> either, there's nothing to compact. The data tier would be no different from
> the meta tier.
>
> When you say you can reindex the stuff on tmpfs quickly, that means you're
> only reindexing the most recent N emails?
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/