Cyrus database and file usage data

[Bron Gondwana]

On Fri, Jun 10, 2016, at 06:40, Thomas Jarosch via Cyrus-devel wrote:
> Hi Bron,
> 
> Am 08.06.2016 um 08:22 schrieb Bron Gondwana via Cyrus-devel:
> > *THE PLAN[tm]***
> >  
> > For JMAP support, I'm going to discard the existing conversations DB and
> > create a sqlite database per user which contains everything of value. 
> 
> one thing to watch out for with sqlite:
> 
> It doesn't scale easily with multiple processes accessing the same DB.
> The write-lock timeout is short by default and a "modifying"
> query might error out.
> 
> If this per-user DB will be used by multiple processes
> (think multiple JMAP connection in parallel from a web frontend)
> + LMTP delivery at the same time, this might occur sooner or later.
> 
> I've learned about this issue the hard way with a central sqlite DB
> for ActiveSync device management on our horde installation.
> 
> Related topic:
> http://beets.io/blog/sqlite-nightmare.html

Yeah, I know - which is why I've been locking around it with an exclusive
file lock so only one process can hit it at a time.

You'd think that would ruin performance, but I haven't actually had too
much trouble.  The conversations DB is already a per-user exclusive lock
whenever you've got any mailbox open right now.

The only place it really hurts is with long running searches.  Long running
fetch is OK, because we drop all the locks after calculating what to send
and before sending anything.

I'm still undecided about just finding a clusterable database format and
skipping over SQL to go straight there...  going to try sqlite first though.

(I might wind up branching this off before 3.0 - it's a massive change, and
we can't hold 3.0 forever)

Bron. 


-- 
  Bron Gondwana
  brong at fastmail.fm