<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>For sure :)<br></div>
<div> </div>
<div>Just having testing infrastructure that tests murder would go a long way to avoiding that mess again.<br></div>
<div> </div>
<div>The more I think about it, the more having the SAME mailboxes.db for both local and remote data doesn't make sense. We should have a separate central database that the mupdate_activate, etc write to. It can just be a standalone SQL database, or a cluster database, or who cares... the main thing is that only a few of the MBOXLIST commands need to care (because they will return the remote information if needed)<br></div>
<div> </div>
<div>Bron.<br></div>
<div> </div>
<div> </div>
<div>On Sat, Mar 14, 2015, at 09:54 AM, Dave McMurtrie wrote:<br></div>
<blockquote type="cite"><div><div>From my phone, so excuse brevity and top-posting, but Fastmail running murder would be a huge bonus. I not-so-fondly recall the intimate relationship I developed with gdb debugging murder issues when we upgraded from 2.3 to 2.4 :)<br></div>
<div> </div>
<div> </div>
<div><div style="font-size:9px; color:#575757">Sent via the Samsung GALAXY SŪ 5, an AT&T 4G LTE smartphone<br></div>
</div>
<div> </div>
<div> </div>
<div> </div>
<div>
-------- Original message --------<br></div>
<div>
From: Bron Gondwana <brong@fastmail.fm> <br></div>
<div>
Date:03/13/2015 6:50 PM (GMT-05:00) <br></div>
<div>
To: Cyrus Devel <cyrus-devel@lists.andrew.cmu.edu> <br></div>
<div>
Cc: <br></div>
<div>
Subject: What would it take for FastMail to run murder <br></div>
<div> </div>
</div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">So I've been doing a lot of thinking about Cyrus clustering, with the</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
underlying question being "what would it take to make FastMail run a</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
murder". We've written a fair bit about our infrastructure - we use</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
nginx as a frontend proxy to direct traffic to backend servers, and have</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
no interdependencies between the backends, so that we can scale</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
indefinitely. With murder as it exists now, we would be pushing the</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
limits of the system already - particularly with the globally</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
distributed datacentres.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Why would FastMail consider running murder, given our existing</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
nice system?</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
a) we support folder sharing within businesses, so at the moment we are</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
limited by the size of a single slot. Some businesses already push</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
that limit.</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
b) it's good to dogfood the server we put so much work into.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Here are our deal-breaker requirements:</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
1) unified murder - we don't want to run both a frontend AND a backend</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
imapd process for every single connection. We already have nginx,</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
which is non-blocking, for the initial connection and auth handling.</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
2) no table scans - anything that requires a parse and ACL lookup for</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
every single row of mailboxes.db is going to be a non- starter when</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
you multiply the existing mailboxes.db size by hundreds.</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
3) no single-point-of-failure - having one mupdate master which can stop</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
the entire cluster working if it's offline, no thanks.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Thankfully, the state of the art in distributed databases has moved a</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
long way since mupdate was written. We'd have to at least change the</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
mupdate protocol anyway to handle newly added fields, so why not just do</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
away with it and have every server run a local node of a distributed</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
database protocol for its mailboxes.db.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Along with this, we need a reverse lookup for ACLs, so that any one user</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
doesn't ever need to scan the entire mailboxes.db. This might be hooked</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
into the distributed DB as well, or calculated locally on each node.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
And that's pretty much it. There are some interesting factors around</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
replication, and I suspect the answer here is to have either multi-</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
value support or embed the backend name into the mailboxes.db key</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
(postfix) such that you wind up listing the same mailbox multiple</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
times. We already suppress duplicates in the LIST command, so all we</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
need then is logic for choosing the actual master. Rob N has done some</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
work with consul and etcd already at FastMail, and we would use either</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
that or a flag in the distributed DB to drive master choice for backend</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
connection purposes.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
There are a bunch of "nice to have"s on top of this, but I think this</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
would be enough for us to convert our existing standalone servers over</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
to a murder.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Bron.</span></span><br></div>
<div> </div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
-- </span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
Bron Gondwana</span></span><br></div>
<div><span style="font-size:small" class="size"><span style="font-size:10pt" class="size">
brong@fastmail.fm</span></span><br></div>
<div> </div>
</blockquote><div> </div>
<div id="sig567075"><div class="signature">--<br></div>
<div class="signature">Bron Gondwana<br></div>
<div class="signature">brong@fastmail.fm<br></div>
<div class="signature"> </div>
</div>
<div> </div>
</body>
</html>