master-master replication
Evgeniy Kononov
egenius at inbox.ru
Thu Sep 13 12:47:08 EDT 2018
Thank you for your experience.
Best regards!
>Четверг, 13 сентября 2018, 19:26 +05:00 от Michael Menge <michael.menge at zdv.uni-tuebingen.de>:
>
>Quoting Evgeniy Kononov < egenius at inbox.ru >:
>
>> Hi!
>>
>> Thank you for reply.
>> Users can connect to only one server at a time. I move the master
>> server to another hardware and at this time it is necessary for
>> users to use the mail.
>> If this is not a secure configuration, then can I just run
>> "sync_client -A" from the master server, and then switch users to a
>> replica?
>> After that, swap the roles of master-replica between the servers? I'm right ?
>>
>>> We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
>>> multiple
>>> instances on one server with nginx frontends
>>
>> Can you give an example of the configuration?
>
>Sure,
>
>first of some background Infos:
>
>We recently switched from Cyrus 2.4.20 on SLES 11 SP4 to Cyrus 3.0.8
>on RHEL 7.5 so consult
>the man pages for your version.
>
>Our Mailserver are running as 6 KVM VMs (RHEV) with 20 GB Ram, 8 Cores each on
>two locations. We have a total of ~44000 accounts, ~457000 Mailboxes,
>and 2x6.5 TB Mails
>
>Each server is running 3-4 instances. One frontend, two backend/replic
>and on one of the servers the cyrus mupdate master. Each Server on one
>location is paired with one server on the other location for replication
>so in normal operation one backend on server A replicates to a replic on
>server B and the backend on server B replicates to the replica on server A.
>
>Keepalived and ipvs loadbalancer distribute the the load to the
>frontend servers.
>We use a private subnet for our backend and replic und mupdate instances and a
>service ip address for the frontends.
>
>We move the ip address with the role, so that ma01.mail.localhost on server A
>replicate to sl01.mail.localhost on server B. But if we need to switch
>to the replic
>we will start it with ma01.mail.localhost on server B
>
>Keeping the master instance for mailbox on the same IP is important,
>because updating the
>location for all mailboxes in the mupdate master would take to long.
>(the mupdate protocol
>knows nothing about replication)
>
>
>The main trick to run multiple instances on one server is to use
>different cyrus.conf
>and imapd.conf files for each instance. We use cyrus_INSTANCE.conf and
>imapd_INSTANCE.conf
>where INSTANCE is replaced by mu for mupdate, fe for the frontend, be
>for the first
>backend/replic and re of the second backend/replic
>
>The choosing of "be" and "re" was not the best as it is easily
>confused with the role
>in wich each of these instances can run.
>
>The masterproces is started with "master -C /etc/imapd_INSTANCE.conf
>-M /etc/cyrus_INSTANCE.conf -p /var/run/cyrus_instance.pid"
>and in the cyrus_INSTANCE.conf you must also use "-C
>/etc/imapd_INSTANCE.conf" service, start and event
>"cmd" so that the correct conf file is used. For services you also
>have to configure "listen="
>so that each instance has its own ip to listen on as only one process
>can listen on 0.0.0.0 for each port.
>In the imapd_INSTANC.conf many directories must be configured.
>
>We generate the conf files from templates. Where TYPE = INSTANCES
>Here are the main parts of our templates
>
>
>========== Cyrus Master ============
># cyrus_@@TYPE@@.conf
># Template MD5SUM: @@MD5SUM@@
>
>START {
> @@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
> @@TYPE@@mupdatepush cmd="ctl_mboxlist -m -a -C /etc/imapd_@@TYPE@@.conf"
> @@TYPE@@idled cmd="idled -C /etc/imapd_@@TYPE@@.conf"
>}
>
>SERVICES {
> @@TYPE@@imap cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
> @@TYPE@@imaps cmd="imapd -U 50 -s -C
>/etc/imapd_@@TYPE@@.conf" listen="@@HOSTNAME@@:imaps" prefork=1
>maxfds=1024
> @@TYPE@@pop3 cmd="pop3d -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:pop3" prefork=1 maxfds=1024
> @@TYPE@@pop3s cmd="pop3d -s -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:pop3s" prefork=1 maxfds=1024
> @@TYPE@@sieve cmd="timsieved -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:sieve" prefork=0 maxfds=1024
> @@TYPE@@lmtp cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:lmtp" prefork=1 maxfds=1024
> @@TYPE@@lmtpunix cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"
>listen="/srv/cyrus-@@TYPE@@/socket/lmtp" prefork=1 maxfds=1024
>}
>
>EVENTS {
> @@TYPE@@checkpoint cmd="ctl_cyrusdb -c -C
>/etc/imapd_@@TYPE@@.conf" period=30
> @@TYPE@@delprune cmd="cyr_expire -E 3 -X 60 -D 60 -C
>/etc/imapd_@@TYPE@@.conf" at=0100
> @@TYPE@@tlsprune cmd="tls_prune -C /etc/imapd_@@TYPE@@.conf" at=0430
> @@TYPE@@squatter cmd="squatter -C /etc/imapd_@@TYPE@@.conf -i" at=2200
>}
>
>======= Cyrus Replic ==============
># cyrus_@@TYPE@@.conf
># Template MD5SUM: @@MD5SUM@@
>
>START {
> @@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
>}
>
>SERVICES {
> @@TYPE@@syncserver cmd="sync_server -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:csync" prefork=1 maxfds=1024
> @@TYPE@@imap cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"
>listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
>}
>
>EVENTS {
> @@TYPE@@checkpoint cmd="ctl_cyrusdb -c -C
>/etc/imapd_@@TYPE@@.conf" period=30
> @@TYPE@@delprune cmd="cyr_expire -E 3 -X 60 -D 60 -C
>/etc/imapd_@@TYPE@@.conf" at=0100
>}
>
>===============
>
>
> Configuration for Backend/Failover Instance
># Template MD5SUM: @@MD5SUM@@
>servername: @@HOSTNAME@@
>configdirectory: /srv/cyrus-@@TYPE@@
>partition-default: /srv/cyrus-@@TYPE@@
>partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
>metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
>metapartition_files: header index cache expunge squat annotations lock
>dav archivecache
>archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
>archive_enabled: 1
>proc_path: /srv/tmpfs/proc-@@TYPE@@
>mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
>defaultpartition: ssd
>admins: XXX
>
>mupdate_server: @@MUPDATEHOSTNAME@@
>mupdate_port: 3905
>mupdate_authname: XXX
>mupdate_password: XXX
>proxy_authname: XXX
>proxy_password: XXX
>proxyservers: XXX
>
>allowallsubscribe: 1
>
>sync_host: @@SYNCHOST@@
>sync_authname: XXX
>sync_password: XXX
>sync_port: 2005
>guid_mode: sha1
>sync_log: 1
>sync_shutdown_file: /srv/cyrus-@@TYPE@@/sync/shutdown
>
>sievedir: /srv/cyrus-@@TYPE@@/sieve
>sieve_extensions: fileinto reject vacation imapflags notify include
>envelope body relational regex subaddress copy
>sieve_maxscriptsize: 150
>
>syslog_prefix: @@TYPE@@
>
>============== Imapd Replic ===============
># Configuration for Slave (Replica) Instance
># Template MD5SUM: @@MD5SUM@@
>servername: @@HOSTNAME@@
>configdirectory: /srv/cyrus-@@TYPE@@
>partition-default: /srv/cyrus-@@TYPE@@
>partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
>metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
>metapartition_files: header index cache expunge squat annotations lock
>dav archivecache
>archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
>archive_enabled: 1
>
>proc_path: /srv/tmpfs/proc-@@TYPE@@
>mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
>defaultpartition: ssd
>admins: XXX
>
>allowusermoves: 1
>allowallsubscribe: 1
>
>proxy_authname: XXX
>proxy_password: XXX
>proxyservers: XXX
>
>sievedir: /srv/cyrus-@@TYPE@@/sieve
>sieve_extensions: fileinto reject vacation imapflags notify include
>envelope body relational regex subaddress copy
>sieve_maxscriptsize: 150
>
>sasl_pwcheck_method: saslauthd
>sasl_mech_list: plain login
>allowanonymouslogin: no
>syslog_prefix: @@TYPE@@
>=================================
>
>The sync client is started as own service
>
>I hope it helps
>
>Regards
>
> Michael
>
>> Best regards.
>>
>>> Четверг, 13 сентября 2018, 13:22 +05:00 от Michael Menge
>>> < michael.menge at zdv.uni-tuebingen.de >:
>>>
>>> Hi,
>>>
>>> This setup is NOT SUPPORTED and WILL BREAK if the replication process
>>> is triggered
>>> from the wrong server (user is active on both servers, user switched
>>> from one server
>>> to the other while the sync-log file is still processed, after split
>>> brain) and
>>> some mailboxes have been subscribed, renamed created deleted.....
>>>
>>> Also there is the risk of a race condition with subscriptions, if a
>>> user subscribes
>>> to multiple folders, the first will trigger a sync from A to B, but as
>>> the folder
>>> is subscribed on B it will trigger a sync from B to A, witch can
>>> undo the next
>>> folder subscription.
>>>
>>> These are only some cases that came to my mind. There will be more
>>> cases and it
>>> will be hard to debug. So DON'T DO IT!
>>>
>>> What we do is, that we have distributed our users between multiple
>>> instances, and each server is running one instance as master and one other
>>> as replic. In case of failure or maintenance we stop the master
>>> instance, and
>>> promote the corresponding replic and configure them so that they will sync
>>> them back. If the old master is up to date we switch them back.
>>>
>>> We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
>>> multiple
>>> instances on one server with nginx frontends
>>>
>>> Regards,
>>>
>>> Michael
>>>
>
>
>--------------------------------------------------------------------------------
>M.Menge Tel.: (49) 7071/29-70316
>Universität Tübingen Fax.: (49) 7071/29-5912
>Zentrum für Datenverarbeitung mail:
>michael.menge at zdv.uni-tuebingen.de
>Wächterstraße 76
>72074 Tübingen
>
--
Evgeniy Kononov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180913/a62b89b6/attachment-0001.html>
More information about the Info-cyrus
mailing list