Clustering and replication

[Tom Samplonius]

----- Wesley Craig <wes at umich.edu> wrote:
> On Jan 26, 2007, at 3:07 AM, Tom Samplonius wrote:
> > ----- Janne Peltonen <janne.peltonen at helsinki.fi> wrote:
> >> As a part of our clustering Cyrus system, we are considering using
> >> replication to prevent a catastrophe in case the volume used by the
> >> cluster gets corrupted. (We'll have n nodes each accessing the  
> >> same GFS,
> >> and yes, it can be done, see previous threads on the subject.)
> >
> >   I really doubt this.  Even if GFS works the way it says it does, 
> 
> > Cyrus does not expect to see other instances modifying the same  
> > message, and does not lock against itself.
> 
> Yes it does.  How else do you suppose two users reading the same  
> shared mailbox might work?  They aren't all running through one
> imapd.

  Within a single server, this works, but Cyrus is not aware of other imapd's that are running on other servers.

  It just isn't the imapd, but the sharing of the database files too.  Can you have writers to a BerkeleyDB database on different servers?  I don't think this works on NFS, let alone GFS.  Getting mmap() to over GFS is a hard problem, and would have to be very slow if it maintained the same semantics as a file on a local disk.

> >> Now the workings of the replication code aren't completely clear  
> >> to me.
> >> It can do things like collapse multiple mailbox changes into one  
> >> and so
> >> on. But is it in some way dependent on there being just one cyrus-
> 
> >> master
> >> controlled group of imapd processes to ensure that all changes to the
> >> spool (and meta) get replicated? Or does the replication code  
> >> infer the
> >> synchronization commands from changes it sees on the spool,  
> >> independent
> >> of the ongoing imap connections? That is, do I have to have n replica
> >> nodes, one for each cluster node? Or don't I?
> >
> >   The Cyrus master builds a replication log as changes are made by 
> > imapd, pop3d, and lmtpd.  The log contents are pushed to the  
> > replica.  The master and replica both have copies of all data,  
> > within independent message stores.
> 
> Close.  imapd, pop3d, lmtpd, and other processes write to the log.   
> The log is read by sync_client.  This merely tells sync_client what  
> (probably) has changed.  sync_client roll up certain log items, e.g., 
> it may decide to compare a whole user's state rather than just  
> looking at multiple mailboxes.  Once it decides what to compare, it  
> retrieves IMAP-like state information from sync_server (running on  
> the replica) and pushes those changes that are necessary.

  And this exposes the big weakness with Cyrus syncing:  there is only a single sync_client, and it is very easy for it get behind.

> For your situation, Janne, you might want to explore sharing the sync 
> directory.  sync_client and sync_server have interlock code, tho I  
> haven't reviewed it for this specific scenario.

  Since the sync directory is specific to the master server, why would you share it?

  Unless, you want to have multiple Cyrus server all pretend to be the master, and log all of their changes to the same sync log.  You would probably hit the sync_client bottleneck pretty fast this way.

  Plus, there would be a lot of contention on the sync logs if multiple servers are appending records to the same file.  GFS is not fast.

> :wes

Tom