Clustering and replication

[Bron Gondwana]

On Mon, Jan 29, 2007 at 04:04:36PM -0800, Tom Samplonius wrote:
>   sync_client prints errors from time to time, but most seem harmless.  It certainly does not print anything like "Exiting...", when it decides to quit.  I don't really know which log lines are bad, or not.  What do you consider a "recognized" log line?  

  next if /\d lockers|Aborting locker/;
  next if m/last message repeated \d+ times/;
  next if m/cyrus_backupd/; # backupd isn't per-slot
  # Ignore errors from cyr_expire run on mailboxes that
  #  don't really exist
  next if /cyr_expire/ && /IOERROR: opening.*cyrus\.header/;

And the code for the sync_client errors:

  # Replication errors
  if (m'Error in do_sync') {
    eval { MailApp::Admin::Actions::NotifyAdmins('email', "cyrus replication error - $HN/$Slot", "cyrus replication error - $HN/$Slot - $_"); };
    next;
  }

>   In my case, sync_client quits three to four times a day.

What version of Cyrus are you running?  We're running a bunch
of patches on top of vanilla Cyrus CVS (post 2.3.7), but even
2.3.7 is pretty good.  2.3.8 should fix most of the known
issues - I think there's still something with renaming Inbox
(creates two folders with the same unique id) - I'm part way
through working on a patch for that.

If you have something corrupted that's breaking sync_client then
every time you try to sync the same mailbox it will break - you
really do need to find the cause.

> > * the system monitoring scripts do a 'du -s' on the sync directory every
> >   2 minutes and store the value in a database so our status commands can
> >   see if any store is behind (the trigger for noticing is 10kb, that's a
> >   couple of minutes worth of log during the U.S. day).  This also emails
> >   us if it gets above 100kb (approx 20 mins behind)
> 
>   And what do you do if it gets behind?  I have three Cyrus groups right now, that are never going to catch up.  They log about 20KB in 20 minutes, so the update rate is not that high.  The machines are dedicated, and the replicas aren't doing anything.  tcpdump confirms that there is traffic to the replica, but the entire sync_client is so opaque it is hard to see what it is doing.  So sync_client can't keep up at all, and since it also quits from time to time, it gets even worse.

Reduce the load on the affected servers until they can cope.  This is usually
done by moving users to a less loaded server, but could also be done short-term
by failing some masters off that host.

>   I'm planning to hack the log, and add some logging to sync_client, particularly to find the number of records per second it is able to process.  And then maybe someway to find why it quits all the time.

That's a pretty pointless metric, since a record could be as little as
"Update the seen state on this one message" or as much as "append this
50Mb email that just arrived", or in an extreme case "synchronise this
entire 10Gb account that just got copied in".
 
>   Either that, or my only alternative is to switch to using DRBD to sync the filesystem to a standby server.

That's an alternative.  We were too worried about filesystem corruption
being copied over to choose it.  Plus, we can monitor for correctness up
a the Cyrus layer now.

> > * a "monitorsync" process that runs from cron every 10 minutes and reads
> >   the contents of the sync directory, comparing any log-(\d+) file's PID
> >   with running processes to ensure it's actually being run and tries to
> >   sync_client -r -f the file if there isn't one.  It also checks that
> >   there is a running sync_client -r process (no -f) for the store.
> 
>   Wow, a lot of protection to protect against sync_client just exiting.  sync_client isn't very big, so it shouldn't be that hard to find the different places that it exits, and fix them?

We do that too.  Meanwhile, we still want this monitoring.  There was
(until Linux 2.6.19.2) a bug in the linux kernel which caused a race
condition in MMAP flushing to disk, and is probably responsible for most
of the occasional database corruption we've seen.  What a pain, and it's
only been discovered recently.  Bah.  These things will cause sync_client
to bail out because it's hit an unrecoverable corruption.  We also want to
detect that.

Basically, I think that no matter how good a system is, it's a lot more
stable with things watching to make sure it's working correctly too.
I prefer the "defense in depth" than trying to make one thing perfect.

Especially since different people within our team wrote these different
tools, so hopefully between us we didn't overlook anything!

> > * a weekly "checkreplication" script which logs in as each user to both
> >   the master and replica via IMAP and does a bunch of lists, examines,
> >   even fetches and compares the output to ensure they are identical.
> > 
> > Between all that, I'm pretty comfortable that replication is correct and
> > we'll be told if it isn't.  It's certainly helped us find our share
> > of issues with the replication system!
> 
>   Well, I know our replicas are out of sync, so we just don't use them.  I just hope the master's don't fail.  Each pair has about 30,000 accounts, and about 300GB of online mail.  

We have multiple (in the teens) of this size stores per machine.  External
SCSI attached SATA RAID units with small high-speed disks in RAID1 for meta
data and much larger slower disks in RAID5 for email files.  By having
multiple masters and replicas per machine we reduce the hit on the "replica
server" if we have to shut down one machine.  Also, this means we can move
across to the replicas and shut one machine down compeletely without any
loss of service (we'll get some emails before long telling us replication=
is behind of course!)

>   And it seems like the multiple exit points in sync_client mean that there are significant bugs in sync_client still.  And since re-starting sync_client on the same sync log appears to work, it means that there is something rather wrong there.

Ok - usually that doesn't work for us.  I really think you're probably
running a more buggy sync_client than us.  Our patches are at:

http://cyrus.brong.fastmail.fm/ - but there aren't many left that are
sync_client related, Ken has already merged most of our fixes upstream.

Bron.