Merging multiple servers into one with partitions

[Bron Gondwana]

On Wed, Sep 05, 2007 at 01:09:30PM -0500, Paul Engle wrote:
> Three independent cyrus servers, each with just a default partition, 
> connected to a SAN. Each server has its own lun mounted as the imap spool 
> partition and things work great.

Have you considered putting the mailboxes.db on the SAN partitions
as well?  It may be that performance isn't good enough, but then
you get more frequent mirroring.

Are you aware that everything else under configdirectory (user/ sieve/
quota/) is essential as well - otherwise you lose seen state,
subscriptions, quotas, etc)  You can't just have the partitions on
the SAN.
 
> A mirrored copy of the SAN data at a remote location, with the same lun 
> partitioning. & a single cyrus server connected to that storage. This 
> server will have three partitions defined in the cyrus config, one for each 
> of the lun mirrors that is normally controlled by the production servers. 
> On an hourly basis, when the mailboxes.db is dumped to text on each of the 
> production servers, it will also be shipped over to the DR server. A script 
> there will munge them, replacing the 'default' partition in each file with 
> the partition name appropriate for that chunk of the mirror, and then merge 
> them into a single input file.

Wow.  Funky.  How are you doing the merge?  
 
> In the event of disaster and the production servers & storage are 
> unavailable, the plan is that the DR server will mount the three luns and 
> the munged/merged mailboxes.db will be imported. The single server will 
> serve the community until such time as the production servers are again 
> available.
> 
> So, my questions:
> 
> 1) Is this just absolutely crazy to even attempt? :)

Almost, but I'll detail another approach in a second.

> 2) I've never played with a cyrus server having multiple partitions. I 
> noticed in some testing that when I use the mkimap tool on the DR server to 
> set up the cyrus framework, the partition directories contain a "sync." 
> directory. This directory doesn't exist on the single-partition servers 
> we're using now. Is that going to pose a problem when the mirrors of those 
> partitions are all mounted on the DR server?

The sync. directories won't hurt.  They're just there for replication.
You upgraded your production systems from an earlier cyrus without
re-running mkimap, didn't you?

> 3) If the answer to #2 is 'yes', then can I safely create an empty "sync." 
> directory on the current production servers so that it will be there for 
> use on the DR server?

That's fine too - cyrus won't even notice it's there unless you do
replication.  Do make it owned by user cyrus if you create it though
so it doesn't confuse you later!

> Any advice/answers are welcome. I know this setup is far from optimal, but 
> it's what we have to work with.

My suggestion - run three instances of cyrus on DR server.  You'll need
three separate IP addresses, but they're pretty cheap in private-space.
Actually, you don't even need that really, just three different ports.
The only thing we've found with forced port numbers is sync_server
(yeah, I should write a patch for that)

I would put _everything_ on the SAN, and make the paths the same on both
servers if you could (even the imapd.conf and cyrus.conf files) - so any
changes are automatically replicated.

Failing that, use a templating system so that your config files are
always consistent at both sites (we use templating extensively, with
everything stored in subversion and "make install" to update each
machine.  Changes by hand are frowned upon)

So you would have:

/usr/cyrus/bin/master -d -C /s1disk/imapd.conf -M /s1disk/cyrus.conf
/usr/cyrus/bin/master -d -C /s2disk/imapd.conf -M /s2disk/cyrus.conf
/usr/cyrus/bin/master -d -C /s3disk/imapd.conf -M /s3disk/cyrus.conf

And these files would contain:

/s1disk/imapd.conf:
configdirectory: /s1disk/conf
defaultpartition: default
partition-default: /s1disk/data

/s2disk/imapd.conf:
configdirectory: /s2disk/conf
defaultpartition: default
partition-default: /s2disk/data

...

/s1disk/cyrus.conf:
 imap  cmd="imapd -C /s1disk/imapd.conf listen="10.10.10.1:143" 

/s2disk/cyrus.conf:
 imap  cmd="imapd -C /s1disk/imapd.conf listen="10.10.10.2:143" 

...

Obviously, I'm only providing small snippets of config here, but
hopefully it's enough for you to understand what I'm showing.

This also means that you can disaster-recover just a single server
with this technique if the other two are working fine.

Regards,

Bron.