Implement Cyrus IMAPD in High Load Enviromment

[Bron Gondwana]

On Wed, Sep 30, 2009 at 01:01:28AM -0400, Brian Awood wrote:
> On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote:
> >
> > Possibly the secret is that we use IPAddr2 from linux-ha to force
> > ARP flushes, and we transfer the primary IP address between
> > machines, so nothing else needs to know - we just shut down one end
> > and bring up the other with the IP and it's all good.
> 
> Our primaries and replicas are located in different data centers, and 
> since we have not control over how the network is subdivided it's 
> impossible for them to take the same IPs.  

Yeah, fair enough.
 
> > Our process is:
> >
> > a) check there are less than 10kb of files in $conf/sync/ - else
> > abort b) shut down the master (host A)
> > c) run sync_client -f $file on each file in $conf/sync (if any)
> > c2) (if any sync fails, restart the master (host A))
> > d) shut down the replica (host B)
> > e) update the database with the new master location
> > f) start up the replica (host A)
> > g) start up the master (host B)
> >
> > This means replication starts immediately, because the replica is
> > already there when the master starts.
> 
> So you just immediately start replicating back to a host (or site) 
> that just failed?  How does that work?

We don't usually "fail" as such.  We're transferring the master role
to a different machine.  Generally you have some advance warning
something bad is happening (like, a single drive in a RAIDset fails)
and transition the master to the less-risky location until the RAID
has rebuilt.

Or you're doing maintainence on the machine that had the master role.

Sure - in a host death situation there's a "force" mode which just
does the "host B" parts.  Then you have to figure out what needs
fixing semi-manually afterwards.  We don't have a cleaner-upperer
yet.

We do have "checkreplication" though, which does a pretty good job
of finding what's wrong between the two machines.
 
> We have a third level of machines that we sync to, in an out of band 
> process, but the data is stored exactly the same way so we can start 
> replicating to them immediately.  So even if a entire data center 
> failed, we can still be running a fully replicated service with 
> almost no downtime visible to users.

Yeah, that would be nice.  We don't have a second datacentre at the
moment.  We're planning to get one running at some point, at least
for the higher-level of paying customers!

Bron.