What would it take for FastMail to run murder

[Bron Gondwana]

On Sat, Mar 14, 2015, at 11:22 PM, Vladislav Bogdanov wrote:
> 14.03.2015 01:50, Bron Gondwana wrote:
> > So I've been doing a lot of thinking about Cyrus clustering, with the
> > underlying question being "what would it take to make FastMail run a
> > murder".  We've written a fair bit about our infrastructure - we use
> > nginx as a frontend proxy to direct traffic to backend servers, and have
> > no interdependencies between the backends, so that we can scale
> > indefinitely.  With murder as it exists now, we would be pushing the
> > limits of the system already - particularly with the globally
> > distributed datacentres.
> 
> Btw (as you speak about clusters), I've developed a Proof-of-the-concept 
> for a cyrus-imapd cluster a long ago using pacemaker as a cluster 
> resource manager. There are many things happened to the linux clustering 
> after that, including remote-node support in the pacemaker, so that 
> concept may be reworked to be even more perfect and scalable.

Hmm, interesting.

> The only 
> thing I did not like that time is that cyrus replication was a bit weak 
> to detect changes after a rolling multi-node failure (node1 goes down, 
> node2 takes over the replica, node2 goes down, node1 goes up and changes 
> made to node2 during node1 was down are lost). Please drop me a note (or 
> just post here as I'm a long time silent reader) if you're interested in 
> making cyrus-imapd rock-solid from the ha-clustering perspective and 
> need some guidance in that so I'll share more details.

So we're a lot closer to good there - the only thing missing is the mailbox level
MODSEQs and tombstones for mailbox listing matches.  Cyrus replication is
pretty close too very nice.

I've love to know how you hooked it all up, it's always interesting to see what
other people do with the code.

Cheers,

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm