Funding Cyrus High Availability

[David Lang]

There are many ways of doing High Availability. This is an attempt to 
outline the various methods with the advantages and disadvantages. Ken and 
David (and anyne else who has thoughts on this) please feel free to add to 
this. I'm attempting to outline them roughly in order of complexity.

1. Active->Slave replication with manual failover

   This is where you can configure one machine to output all changes to a 
local daemon and another machine to implement the changes that are read 
from a local daemon.

   Pro:
    simplist implementation, since it makes no assumptions about how you 
are going to use it, it also sets no limits on how it is used.

    This is the basic functionality that all other variations will need so 
it's not wasted work no matter what is done later

    allows for multiple slaves from a single master

    allows for the propogation traffic pattern to be defined by the 
sysadmin (either master directly to all slaves or a tree-like propogation 
to save on WAN bandwidth when multiple slaves are co-located

    by involving a local daemon at each server there is a lot of 
flexibility in exactly how the replication takes place.
      for example you could
         use netcat as your daemon for instant transmission of the 
messages
         have a daemon that caches the messages so that if the link 
drops the messages are saved
         have a daemon that gets an acknowlegement from the far side that 
the message got through
         have a daemon that batches the messages up and compresses them for 
more efficiant transport
         have a daemon that delays all messages by a given time period to 
give you a way to recover from logical corruption without having to go to 
a backup
         have a daemon that filters the messages (say one that updates 
everything except it won't delete any messages so you have a known safe 
archive of all messages)
         etc

   Con:
    since it makes no assumptions about how you are going to use it, it 
also gives you no help in useing it in any particular way


2. Active->Slave replication with automatic failover

   This takes #1, limits it to a pair of boxes and through changes to 
murder or other parts of cyrus will swap the active/slave status of the 
two boxes

   Pro:
    makes setting up of a HA pair of boxes easier

    increases availability by decreasing downtime

   Con:
    this functionality can be duplicated without changes to cyrus by the 
use of an external HA/cluster software package.

    Since this now assumes a particular mode of operation it starts to 
limit other uses (for example, if this is implemented as part of murder 
then it won't help much if you are trying to replicate to a DR datacenter 
several thousand miles away).

    Split-brain conditions are the responsibility of cyrus to prevent or 
solve. These are fundamentaly hard problems to get right in all cases


3. Active->Slave replication with Slave able to accept client connections

   This takes #1 and then further modifies the slave so that requests that 
would change the contents of things get relayed to the active box and then 
the results of the change get propogated back down before they are visable 
to the client.

   Pro:
    simulates active/active operation although it does cause longer delays 
when clients issue some commands.

    use of slaves for local access can reduce the load on the master 
resulting in higher performance.

    can be cascaded to multiple slaves and multiple tiers of slaves as 
needed

    in case of problems on the master the slaves can continue to operate as 
read-only servers providing degraded service while the master is fixed. 
depending on the problem with the master this may be very preferable to 
having to re-sync the master or recover from a split-brain situation

   Con:
    more extensive modifications needed to trap all changes and propogate 
them up to the master

    how does the slave know when the master has implemented the change (so 
that it can give the result to the client)

    raises questions about the requirement to get confirmation og all 
updates before the slave can respond to the client (for example, if a 
slave decides to read a message that is flagged as new should the slave 
wait until the master confirms that it knows the message has been read 
before it gives it to the client, or should it give the message to the 
client and not worry if the update fails on the master)

    since the slave needs to send updates to the master the latency of the 
link between them can become a limiting factor in the performance that 
clients see when connecting to the slave

4. #3 with automatic failover

   Since #3 supports multiple slaves the number of failover senerios grow 
significantly. you have multiple machines that could be the new master and 
you have the split-brain senerio to watch out for.

   Pro:
    increased availability by decreasing failover time

    potentially easier to setup then with external clustering software

   Con:
    increased complexity

    runs the risk of breaking some deployment senerios in an attempt to 
simplify others

5. Active/Active

   designate one of the boxes as primary and identify all items in the 
datastore that absolutly must not be subject to race conditions between 
the two boxes (message UUID for example). In addition to implementing the 
replication needed for #1 modify all functions that need to update these 
critical pieces of data to update them on the master and let the master 
update the other box.

   Pro:
    best use of available hardware as the load is split almost evenly 
between the boxes.

    best availability becouse if there is a failure half of the clients 
won't see it at all

   Con:
    significantly more complex then the other options.

    behavior during a failure is less obvious

    split-brain recovery is not straightforward and if automatic failover 
is active the sysadmin will have no option to have things degraded 
slightly while a problem is fixed

    depending on the implementation this may be very sensitive to network 
latency between the machines and could be very suitable for working with 
machines in the same datacenter, but worthless for machines thousands of 
miles apart.

6. active/active/active/...

   Take #5 and extend the idea to more then a pair of boxes. this makes the 
updates more complex to propogate (they now need to be sent to every other 
machine in the cluster)

   Pro:
    better load balancing then #5

    allows for the ability to have a HA pair in a primary location and a 
backup in a remote location (i.e. your main HQ has two boxes, but your 
disaster recovery center has one as well)

   Con:
    the complexity goes up significantly when you shift from 2 to n boxes 
in a cluster.

    the bandwidth required for updates increases by a factor of roughly n!

    significantly more split-brain senerios become possible and need to be 
accounted for.



-------------------------------------------------------------------------

while #6 is the ideal option to have it can get very complex

personally I would like to see #1 (with a sample daemon or two to provide 
basic functionality and leave the doors open for more creative uses) 
followed by #3 while people try and figure out all the problems with #5 
and #6

there are a lot of senerios that are possible with #1 or #3 that are not 
possible with #5 and very little of the work needed to release #1 and #3 
as supported options is not work that needs to be done towards #5/6 anyway 
(the pieces need to be identified in the code and hooks put in place in 
the code at those locations. the details of the hooks will differ slightly

David Lang

  -- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html