painful mupdate syncs between front-ends and database server

[Brian Awood]

On Monday 19 October 2009 @ 16:38, Michael Bacon wrote:
>
> I say mostly because while most of the times the thing handles our
> 80,000 users and 14,000+ simultaneous connections like a champ,
> some of the time, we get some extreme pain, mostly due to syncs
> between the MUPDATE master and the front-end servers.

We have similar usage levels, but don't have any issues with proxies 
and MUPDATE.  What is the load on your mupdate master like?  When a 
proxy mupdate first connects it should get a dump of the database 
then sit and wait for any changes to be sent to it.  If the load on 
your mupdate master is high, have you tried setting;  
mupdate_config: unified
on your proxies?

> I suppose this is Fastmail and others ripped out the proxyd's and
> replaced them with nginx or perdition.  Currently we still support
> GSSAPI as an auth mechanism, which kept me from going that
> direction, but given the problems we're seeing, I'd be open to
> architectural suggestions on either how to tie perdition or nginx
> to the MUPDATE master (because we don't have the back-ends split
> along any discernable lines at this point), or suggestions on how
> to make the master-to-frontend propagation faster or less painful.
>

For comparison, we also support GSSAPI.  We have 3 IPs behind our mail 
hostname, each IP is a hardware load balancer with 2 machines behind 
it.  The system load on each of the 6 machines is generally not much 
more than 1.  Before the load balancers, when we just used a muti-A 
record we would tend to have some proxyd related load issues because 
mail clients tend to prefer the lowest IP in a set.

-Brian