Implement Cyrus IMAPD in High Load Enviromment
brong at fastmail.fm
Wed Sep 30 02:30:05 EDT 2009
On Wed, Sep 30, 2009 at 01:01:28AM -0400, Brian Awood wrote:
> On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote:
> > Possibly the secret is that we use IPAddr2 from linux-ha to force
> > ARP flushes, and we transfer the primary IP address between
> > machines, so nothing else needs to know - we just shut down one end
> > and bring up the other with the IP and it's all good.
> Our primaries and replicas are located in different data centers, and
> since we have not control over how the network is subdivided it's
> impossible for them to take the same IPs.
Yeah, fair enough.
> > Our process is:
> > a) check there are less than 10kb of files in $conf/sync/ - else
> > abort b) shut down the master (host A)
> > c) run sync_client -f $file on each file in $conf/sync (if any)
> > c2) (if any sync fails, restart the master (host A))
> > d) shut down the replica (host B)
> > e) update the database with the new master location
> > f) start up the replica (host A)
> > g) start up the master (host B)
> > This means replication starts immediately, because the replica is
> > already there when the master starts.
> So you just immediately start replicating back to a host (or site)
> that just failed? How does that work?
We don't usually "fail" as such. We're transferring the master role
to a different machine. Generally you have some advance warning
something bad is happening (like, a single drive in a RAIDset fails)
and transition the master to the less-risky location until the RAID
Or you're doing maintainence on the machine that had the master role.
Sure - in a host death situation there's a "force" mode which just
does the "host B" parts. Then you have to figure out what needs
fixing semi-manually afterwards. We don't have a cleaner-upperer
We do have "checkreplication" though, which does a pretty good job
of finding what's wrong between the two machines.
> We have a third level of machines that we sync to, in an out of band
> process, but the data is stored exactly the same way so we can start
> replicating to them immediately. So even if a entire data center
> failed, we can still be running a fully replicated service with
> almost no downtime visible to users.
Yeah, that would be nice. We don't have a second datacentre at the
moment. We're planning to get one running at some point, at least
for the higher-level of paying customers!
More information about the Info-cyrus