load balancing at fastmail.fm

[Bron Gondwana]

On Thu, May 10, 2007 at 09:49:01AM -0400, Nik Conwell wrote:
> 
> On Jan 12, 2007, at 10:43 PM, Rob Mueller wrote:
> 
> >Yep, this means we need quite a bit more software to manage the  
> >setup, but now that it's done, it's quite nice and works well. For  
> >maintenance, we can safely fail all masters off a server in a few  
> >minutes, about 10-30 seconds a store. Then we can take the machine  
> >down, do whatever we want, bring it back up, wait for replication  
> >to catch up again, then fail any masters we want back on to the  
> >server.
> 
> Just curious how you do this - do you just stop the masters and then 
> change the proxy to point to the replica?  Webmail users shouldn't  
> notice this but don't the desktop IMAP clients notice?

We use IPAddr2 from linux-ha to bind the master IP address and replica
IP address to each machine based on the database entry saying which
slot is the master.  That way we don't need to change anything else,
you just connect to the master IP address.

It also means every slot can just bind to the standard ports on its
IP address.

As you can imagine, there's a lot of templating and custom config and
init scripts going on here - but it all works nicely once you're set
up!

The failover scripts also run sync_client on leftover log files and
other consistency checks.

Bron.