DNS load balancing

[Andy Bennett]

Hi,

> I'm setting up a two-machine cyrus cluster using OCFS2 over DRBD. The
> setup is working fine, and I'm now considering the load balancing
> options I have.
> 
> I believe the simplest option would be to simply rely on DNS load
> balancing. However, for this to work, I need to consider what happens
> when one of the cluster nodes fails, regarding the clients which access
> the cluster:
> 
>  - Will IMAP and POP3 clients automatically try the second DNS entry if 
>    connecting to the first one fails?
>  - Will the LMTP client also do this? I'm using the postfix LMTP client 
>    for delivery.
> 
> Does any one have experience with this kind of setup?

I've not done this with Cyrus servers but to make DNS load balancing 
like this "failover safe" there are a couple of solutions.

I guess in DNS you have something like the following:

-----
imap  A a.b.c.x
imap  A a.b.c.y
host1 A a.b.c.x
host2 A a.b.c.y
-----




If you have a couple of spare IPs you can assign things like this instead:

-----
imap  A a.b.c.x
imap  A a.b.c.y
host1 A a.b.c.p
host2 A a.b.c.q
-----

Then use something like wackamole (http://www.backhand.org/wackamole/) 
to ensure that a.b.c.{x,y} are always being listened on by exactly 1 
machine each. a.b.c.{p,q} would be used to admin access to specific 
machines.

This option involves installing and configuring the wackamole software 
on each host.

DNS will give each client an arbitrary IP address. If the server 
listening on that IP address fails wackamole will move the IP from the 
failed host to another. This will cause all the connected clients to 
have to reconnect. New clients that come along whilst a node failure is 
in progress will not experience anything unusual but you might have to 
check the semantics when a new server is added to the pool.




The alternative is to delgate 'imap' as a subdomain in DNS thusly:

-----
imap NS host1
imap NS host2
host1 A a.b.c.p
host2 A a.b.c.q
-----

Then run bind on host1 and host2 to serve the imap zone. On host1 it 
would look like this:

-----
@ A a.b.c.p
-----

and on host2 it would look like this:

-----
@ A a.b.c.q
-----

You should set appropriately short TTLs on imap and the delegated zone.

This will mean that each client will randomly resolve to one of the 
machines. When a machine is down its DNS server will also be down so the 
lookup will timeout in the client's resolver and it will try the other 
NS servers listed in turn until it gets a server that will give it an A 
record.

In failure cases existing clients will wobble for a bit until their 
cache expires and then the connections will have to be reestablished. 
New clients will just take slightly longer to connect in some situations 
but that will (hopefully) pass mostly unnoticed by the users.








Regards,
@ndy

-- 
andyjpb at ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF