<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Wow, interesting. That definitely works, though I'd probably normalise everything to the user ID so that the seen and mailbox events for the same user got the same channel.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">We're looking at similar things for our setup too, either shading or even per user logs with a daemon which farms users out to multiple channels.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">As for when we'd look at a sync daemon: probably next year. We're planning to land uuid based storage soon, which means that renaming users and mailboxes is really fast, then looking at replication channels on top of that would make more sense, because otherwise user renames become tricky.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I'll have a look at the diff when it isn't 11:30pm for me.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Cheers,</div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron</div><div style="font-family:Arial;"><br></div><div>On Thu, Nov 21, 2019, at 18:50, Thomas Cataldo wrote:<br></div><blockquote type="cite" id="qt"><div>Hi,<br></div><div><br></div><div>In our workload, cyrus replication latency is pretty critical as we serve most read requests from the replica.<br></div><div>Having a single network channel between master & replica is a big issue for us.<br></div><div><br></div><div>Trying to improve our latency, we implemented the following approach : instead of writing “channel/log” we write “channel/log.<shard_index>”.<br></div><div>We compute our shard key this way :<br></div><div><br></div><div># cat log.0 <br></div><div>APPEND devenv.blue!user.tom.Sent<br></div><div>MAILBOX devenv.blue!user.tom.Sent<br></div><div><br></div><div># cat log.2 <br></div><div>SEEN tom@devenv.blue 9f799278-a6cd-45b7-9546-0e861d5e15d6<br></div><div>root@bm1804:/var/lib/cyrus/sync/core# cat log.3 <br></div><div>…<br></div><div>APPEND devenv.blue!user.sga<br></div><div>MAILBOX devenv.blue!user.sga<br></div><div><br></div><div>We compute an hashcode of the first argument. We normalize it so devenv.blue!user.tom.Sent and devenv.blue!user.tom have the same hashcode then we “hashcode % shard_count” to figure out which log file to use.<br></div><div>We patched sync_client to add a “-i <shard_index>”. sync_client -i 0 will process log.0 and use log-run.0, etc.<br></div><div><br></div><div>We don’t spawn sync_client from cyrus.conf but we prefer systemd tricks :<br></div><div><br></div><div>/lib/systemd/system/bm-cyrus-syncclient@.service which is a template and we then enable :<br></div><div>systemctl enable bm-cyrus-syncclient@{0..3} to spawn 4 sync_client.<br></div><div><br></div><div><br></div><div>Attached diff of what we changed. <br></div><div><br></div><div>As a side note, our usage forbids moving a mailbox folder into another mailbox (ie. moving user.tom.titi into user.sga.stuff is forbidden in our setup). I guess this approach would be problematic we moving a mailbox subfolder to another mailbox as they might be sharded to separate log files.<br></div><div><br></div><div>Any feedback on this approach ? I read that you planned to turn sync_client into a sync daemon. Any schedule estimate on that ?<br></div><div><br></div><div>Regards,<br></div><div>Thomas.<br></div><div><br></div><div><br></div><div>sync_client systemd configuration template :<br></div><div>/lib/systemd/system/bm-cyrus-syncclient@.service (%i is expanded to 42 by systemd when you enable syncclient@42)<br></div><div>[Unit]<br></div><div>Description=BlueMind Cyrus sync_client service<br></div><div>After=bm-cyrus-imapd.service<br></div><div>PartOf=bm-cyrus-imapd.service<br></div><div>ConditionPathExists=!/etc/bm/bm-cyrus-imapd.disabled<br></div><div><br></div><div>[Service]<br></div><div>Type=forking<br></div><div>Environment=CONF=/etc/imapd.conf<br></div><div>ExecStartPre=/usr/bin/find /var/lib/cyrus/sync -name ‘log*.%i' -type f -exec rm -f {} \;<br></div><div>ExecStart=/usr/sbin/sync_client -C $CONF -t 1800 -n core -i %i -l -r<br></div><div>SuccessExitStatus=75<br></div><div>RemainAfterExit=no<br></div><div>Restart=always<br></div><div>RestartSec=5s<br></div><div>TimeoutStopSec=20s<br></div><div><br></div><div>[Install]<br></div><div>WantedBy=bm-cyrus-imapd.service<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Thomas Cataldo<br></div><div>Directeur Technique<br></div><div><br></div><div>(+33) 6 42 25 91 38<br></div><div><br></div><div>BlueMind<br></div><div>+33 (0)5 81 91 55 60<br></div><div>Hotel des Télécoms, 40 rue du village d'entreprises<br></div><div>31670 Labège, France<br></div><div>www.bluemind.net / https://blog.bluemind.net/fr/<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><b>Attachments:</b><br></div><ul><li>replication_multiplexing.diff<br></li></ul></blockquote><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, Fastmail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div></body></html>