master-master replication

[Michael Menge]

Quoting Evgeniy Kononov <egenius at inbox.ru>:

> Hi!
>
> Thank you for reply.
> Users can connect to only one server at a time. I move the master  
> server to another hardware and at this time it is necessary for  
> users to use the mail.
> If this is not a secure configuration, then can I just run  
> "sync_client -A" from the master server, and then switch users to a  
> replica?
> After that, swap the roles of master-replica between the servers? I'm right ?
>
>> We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses 
>> multiple
>> instances on one server with nginx frontends
>
> Can you give an example of the configuration?

Sure,

first of some background Infos:

We recently switched from Cyrus 2.4.20 on SLES 11 SP4 to Cyrus 3.0.8  
on RHEL 7.5 so consult
the man pages for your version.

Our Mailserver are running as 6 KVM VMs (RHEV) with 20 GB Ram, 8 Cores each on
two locations. We have a total of ~44000 accounts, ~457000 Mailboxes,  
and 2x6.5 TB Mails

Each server is running 3-4 instances. One frontend, two backend/replic
and on one of the servers the cyrus mupdate master. Each Server on one
location is paired with one server on the other location for replication
so in normal operation one backend on server A replicates to a replic on
server B and the backend on server B replicates to the replica on server A.

Keepalived and ipvs loadbalancer distribute the the load to the  
frontend servers.
We use a private subnet for our backend and replic und mupdate instances and a
service ip address for the frontends.

We move the ip address with the role, so that ma01.mail.localhost on server A
replicate to sl01.mail.localhost on server B. But if we need to switch  
to the replic
we will start it with ma01.mail.localhost on server B

Keeping the master instance for mailbox on the same IP is important,  
because updating the
location for all mailboxes in the mupdate master would take to long.  
(the mupdate protocol
knows nothing about replication)


The main trick to run multiple instances on one server is to use  
different cyrus.conf
and imapd.conf files for each instance. We use cyrus_INSTANCE.conf and  
imapd_INSTANCE.conf
where INSTANCE is replaced by mu for mupdate, fe for the frontend, be  
for the first
backend/replic and re of the second backend/replic

The choosing of "be" and "re" was not the best as it is easily  
confused with the role
in wich each of these instances can run.

The masterproces is started with "master -C /etc/imapd_INSTANCE.conf  
-M /etc/cyrus_INSTANCE.conf -p /var/run/cyrus_instance.pid"
and in the cyrus_INSTANCE.conf you must also use "-C  
/etc/imapd_INSTANCE.conf" service, start and event
"cmd" so that the correct conf file is used. For services you also  
have to configure "listen="
so that each instance has its own ip to listen on as only one process  
can listen on 0.0.0.0 for each port.
In the imapd_INSTANC.conf many directories must be configured.

We generate the conf files from templates. Where TYPE = INSTANCES
Here are the main parts of our templates


========== Cyrus Master ============
# cyrus_@@TYPE@@.conf
# Template MD5SUM: @@MD5SUM@@

START {
    @@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
    @@TYPE@@mupdatepush cmd="ctl_mboxlist -m -a -C /etc/imapd_@@TYPE@@.conf"
    @@TYPE@@idled cmd="idled -C /etc/imapd_@@TYPE@@.conf"
}

SERVICES {
    @@TYPE@@imap        cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
    @@TYPE@@imaps       cmd="imapd -U 50 -s -C  
/etc/imapd_@@TYPE@@.conf" listen="@@HOSTNAME@@:imaps" prefork=1  
maxfds=1024
    @@TYPE@@pop3        cmd="pop3d -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:pop3" prefork=1 maxfds=1024
    @@TYPE@@pop3s       cmd="pop3d -s -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:pop3s" prefork=1 maxfds=1024
    @@TYPE@@sieve       cmd="timsieved -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:sieve" prefork=0 maxfds=1024
    @@TYPE@@lmtp        cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:lmtp" prefork=1 maxfds=1024
    @@TYPE@@lmtpunix    cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"  
listen="/srv/cyrus-@@TYPE@@/socket/lmtp" prefork=1 maxfds=1024
}

EVENTS {
    @@TYPE@@checkpoint    cmd="ctl_cyrusdb -c -C  
/etc/imapd_@@TYPE@@.conf" period=30
    @@TYPE@@delprune      cmd="cyr_expire -E 3 -X 60 -D 60 -C  
/etc/imapd_@@TYPE@@.conf" at=0100
    @@TYPE@@tlsprune      cmd="tls_prune -C /etc/imapd_@@TYPE@@.conf" at=0430
    @@TYPE@@squatter      cmd="squatter -C /etc/imapd_@@TYPE@@.conf -i" at=2200
}

======= Cyrus Replic ==============
# cyrus_@@TYPE@@.conf
# Template MD5SUM: @@MD5SUM@@

START {
    @@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
}

SERVICES {
    @@TYPE@@syncserver  cmd="sync_server -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:csync" prefork=1 maxfds=1024
    @@TYPE@@imap        cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
}

EVENTS {
    @@TYPE@@checkpoint    cmd="ctl_cyrusdb -c -C  
/etc/imapd_@@TYPE@@.conf" period=30
    @@TYPE@@delprune      cmd="cyr_expire -E 3 -X 60 -D 60 -C  
/etc/imapd_@@TYPE@@.conf" at=0100
}

===============


  Configuration for Backend/Failover Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache
archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1
proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

mupdate_server: @@MUPDATEHOSTNAME@@
mupdate_port: 3905
mupdate_authname: XXX
mupdate_password: XXX
proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

allowallsubscribe: 1

sync_host: @@SYNCHOST@@
sync_authname: XXX
sync_password: XXX
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-@@TYPE@@/sync/shutdown

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy
sieve_maxscriptsize: 150

syslog_prefix: @@TYPE@@

============== Imapd Replic ===============
# Configuration for Slave (Replica) Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache
archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1

proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

allowusermoves: 1
allowallsubscribe: 1

proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy
sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login
allowanonymouslogin: no
syslog_prefix: @@TYPE@@
=================================

The sync client is started as own service

I hope it helps

Regards

    Michael

> Best regards.
>
>> Четверг, 13 сентября 2018, 13:22 +05:00 от Michael Menge  
>> <michael.menge at zdv.uni-tuebingen.de>:
>>
>> Hi,
>>
>> This setup is NOT SUPPORTED and WILL BREAK if the replication process
>> is triggered
>> from the wrong server (user is active on both servers, user switched
>> from one server
>> to the other while the sync-log file is still processed, after split
>> brain) and
>> some mailboxes have been subscribed, renamed created deleted.....
>>
>> Also there is the risk of a race condition with subscriptions, if a
>> user subscribes
>> to multiple folders, the first will trigger a sync from A to B, but as
>> the folder
>> is subscribed on B it will trigger a sync from B to A, witch can  
>> undo the next
>> folder subscription.
>>
>> These are only some cases that came to my mind. There will be more
>> cases and it
>> will be hard to debug. So DON'T DO IT!
>>
>> What we do is, that we have distributed our users between multiple
>> instances, and each server is running one instance as master and one other
>> as replic. In case of failure or maintenance we stop the master  
>> instance, and
>> promote the corresponding replic and configure them so that they will sync
>> them back. If the old master is up to date we switch them back.
>>
>> We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
>> multiple
>> instances on one server with nginx frontends
>>
>> Regards,
>>
>>     Michael
>>


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen