[RFC] multiplexing cyrus replication with log/log-run sharding & multiple sync_client

[Thomas Cataldo]

Hi,

In our workload, cyrus replication latency is pretty critical as we serve most read requests from the replica.
Having a single network channel between master & replica is a big issue for us.

Trying to improve our latency, we implemented the following approach : instead of writing “channel/log” we write “channel/log.<shard_index>”.
We compute our shard key this way :

# cat log.0 
APPEND devenv.blue!user.tom.Sent
MAILBOX devenv.blue!user.tom.Sent

# cat log.2 
SEEN tom at devenv.blue 9f799278-a6cd-45b7-9546-0e861d5e15d6
root at bm1804:/var/lib/cyrus/sync/core# cat log.3 
…
APPEND devenv.blue!user.sga
MAILBOX devenv.blue!user.sga

We compute an hashcode of the first argument. We normalize it so devenv.blue!user.tom.Sent and devenv.blue!user.tom have the same hashcode then we “hashcode % shard_count” to figure out which log file to use.
We patched sync_client to add a “-i <shard_index>”. sync_client -i 0 will process log.0 and use log-run.0, etc.

We don’t spawn sync_client from cyrus.conf but we prefer systemd tricks :

/lib/systemd/system/bm-cyrus-syncclient at .service which is a template and we then enable :
systemctl enable bm-cyrus-syncclient@{0..3} to spawn 4 sync_client.


Attached diff of what we changed. 

As a side note, our usage forbids moving a mailbox folder into another mailbox (ie. moving user.tom.titi into user.sga.stuff is forbidden in our setup). I guess this approach would be problematic we moving a mailbox subfolder to another mailbox as they might be sharded to separate log files.

Any feedback on this approach ? I read that you planned to turn sync_client into a sync daemon. Any schedule estimate on that ?

Regards,
Thomas.


sync_client systemd configuration template :
/lib/systemd/system/bm-cyrus-syncclient at .service (%i is expanded to 42 by systemd when you enable syncclient at 42)
[Unit]
Description=BlueMind Cyrus sync_client service
After=bm-cyrus-imapd.service
PartOf=bm-cyrus-imapd.service
ConditionPathExists=!/etc/bm/bm-cyrus-imapd.disabled

[Service]
Type=forking
Environment=CONF=/etc/imapd.conf
ExecStartPre=/usr/bin/find /var/lib/cyrus/sync -name ‘log*.%i' -type f -exec rm -f {} \;
ExecStart=/usr/sbin/sync_client -C $CONF -t 1800 -n core -i %i -l -r
SuccessExitStatus=75
RemainAfterExit=no
Restart=always
RestartSec=5s
TimeoutStopSec=20s

[Install]
WantedBy=bm-cyrus-imapd.service





Thomas Cataldo
Directeur Technique 

(+33) 6 42 25 91 38

BlueMind
+33 (0)5 81 91 55 60
Hotel des Télécoms, 40 rue du village d'entreprises
31670 Labège, France
www.bluemind.net / https://blog.bluemind.net/fr/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: replication_multiplexing.diff
Type: application/octet-stream
Size: 23847 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20191121/2db29f64/attachment.obj>
-------------- next part --------------