<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi,
<div class="">
<br class="">
</div>
<div class="">
We are working with a replication enabled cyrus 3.0.8.
</div>
<div class="">
Our replication endpoint is not another cyrus but I don’t think it matters here. I am working on enabling object storage support with our setup.
</div>
<div class="">
<br class="">
</div>
<div class="">
Our replication settiings look like this :
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">sync_log: 1</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">sync_log_channels: core</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span>
<br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_authname: admin0</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_password: xxxxx</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_realm: repl_realm</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_host: 172.16.167.129</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_port: 2501</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_repeat_interval: 0</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">core_sync_try_imap: 0</span>
</div>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">We made a copy of objectstore_dummy.c and simplified it to hardcode the path to /dummy-sds/<message guid> for testing purposes.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">The object storage/archive setup looks like this :</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">object_storage_enabled: 1</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">archive_enabled: 1</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">archive_days: 0</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">archive_maxsize: 0</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
archive_keepflagged: 0
<span style="font-variant-ligatures: no-common-ligatures" class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">From an imap point of view, everything works fine, emails are written through the object storage and are read from there.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""># ls -l /dummy-sds/</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">...</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
-rw------- 1 cyrus mail 1491 Aug 28 11:35 ae3aaefa50d04042bc369e29e9c819ed600d3a03
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
-rw------- 1 cyrus mail 863 Aug 28 11:35 f996f4ad4e4c37c3c2cc852214fdeca778d9c43f
</div></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">The problem occurs when sync_client triggers its replication code.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">From our logs we see:</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: <b class="">creating sql_db /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/message.db</b></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/imap[116313]: <b class="">USAGE <a href="mailto:admin@devenv.blue" class="">admin@devenv.blue</a> user: 0.000000 sys: 0.010090</b></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES devenv.blue!user.admin.Sent devenv.blue!user.tom</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: Delivered: <<a href="mailto:ffe06ca2d52c5d64a0d5401f5f23a29a@devenv.blue" class="">ffe06ca2d52c5d64a0d5401f5f23a29a@devenv.blue</a>> to mailbox: devenv.blue!user.tom</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX devenv.blue!user.admin.Sent</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(180, 36, 25);" class="">
<span style="font-variant-ligatures: no-common-ligatures; color: #000000" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: </span>
<span style="font-variant-ligatures: no-common-ligatures" class=""><b class="">IOERROR: Failed to read file /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/a/user/admin/Sent/4.</b></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: <b class="">USAGE <a href="mailto:tom@devenv.blue" class="">tom@devenv.blue</a> user: 0.000000 sys: 0.011507</b></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX devenv.blue!user.tom</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(180, 36, 25);" class="">
<span style="font-variant-ligatures: no-common-ligatures; color: #000000" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: </span>
<span style="font-variant-ligatures: no-common-ligatures" class=""><b class="">IOERROR: Failed to read file /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/1.</b></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES devenv.blue!user.tom</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/imap[33172]: session initialised for <a href="mailto:admin@devenv.blue" class="">admin@devenv.blue</a></span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Aug 28 11:35:23 bm1804 cyrus/imap[33172]: <b class="">login: bm1804.devenv.blue [172.16.167.129] <a href="mailto:admin@devenv.blue" class="">admin@devenv.blue</a> PLAIN User logged in SESSIONID=<cyrus-33172-1566984923-1-5992269715888192037></b></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><b class=""><br class=""></b></span>
</div></span>
</div>
<div class="">
<br class="">
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">From a protocol point of view we receive (REPL C is what sync_client sends us, REPL S is what we respond):</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2019-08-28 09:35:23,310 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000001]: GET MAILBOXES (devenv.blue!user.admin.Sent devenv.blue!user.tom)</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2019-08-28 09:35:23,326 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000001]: * MAILBOX %(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME devenv.blue!user.admin.Sent SYNC_CRC 583232180 SYNC_CRC_ANNOT 0 LAST_UID 3 HIGHESTMODSEQ 6 RECENTUID 3 RECENTTIME 1566981904 LAST_APP... [truncated]</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2019-08-28 09:35:23,329 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000002]: APPLY RESERVE %(PARTITION bm-master__devenv_blue MBOXNAME (devenv.blue!user.admin.Sent devenv.blue!user.tom) GUID (f996f4ad4e4c37c3c2cc852214fdeca778d9c43f ae3aaefa50d04042bc369e29e9c819ed600d3a03))</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2019-08-28 09:35:23,332 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000002]: * MISSING (f996f4ad4e4c37c3c2cc852214fdeca778d9c43f ae3aaefa50d04042bc369e29e9c819ed600d3a03)
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">OK success</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000003]: APPLY MESSAGE (NIL)</span>
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000003]: OK success
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2019-08-28 09:35:23,337 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000004]: APPLY MAILBOX %(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME devenv.blue!user.admin.Sent SYNC_CRC 3650189183 SYNC_CRC_ANNOT 0 LAST_UID 4 HIGHESTMODSEQ 7 RECENTUID 3 RECENTTIME 1566981904 LAST_APPENDDATE 1566984923 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1566980951 PARTITION bm-master__devenv_blue ACL "admin0<span class="Apple-tab-span" style="white-space:pre"> </span>lrswipkxtecdan<span class="Apple-tab-span" style="white-space:pre"> </span><a href="mailto:E8F1E28F-8778-442B-A581-C6B613CE9555@devenv.blue" class="">E8F1E28F-8778-442B-A581-C6B613CE9555@devenv.blue</a><span class="Apple-tab-span" style="white-space:pre"> </span>lrswipktecdan<span class="Apple-tab-span" style="white-space:pre"> </span>" OPTIONS P ANNOTATIONS (%(ENTRY /specialuse USERID admin@devenv.blue VALUE {5+}{t28.bin})) RECORD (%(UID 4 MODSEQ 7 LAST_UPDATED 1566984923 FLAGS (\Seen) INTERNALDATE 1566984923 SIZE 863 GUID f996f4ad4e4c37c3c2cc852214fdeca778d9c43f)))</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">The problem seems pretty “obvious” : sync_client APPLY RESERVE for the 2 bodies (from Sent folder + the copy for the recipient inbox), the it tries to read the bodies but I imagine the sync_client code is not object-storage enabled correctly and only tries the local filesystem path instead of asking to the object storage.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">It could also be an expected behaviour, as the replica(s) can share the object storage with master and should reply that nothing is missing.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">What is your point of view on that ? Could you give me a point to where in the sync_client code I should look to “object storage”-enable it.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
Quite un-related but we enjoyed reading your stuff about sync improvements and having multiple sync streams between master & replica. If we can help on that we would be happy to.
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Regards,</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Thomas.</span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span>
</div>
<div class="x-disclaimer-1318405639">
<table style="width: 558px;">
<tbody>
<tr>
<td style="width: 553.466px;" colspan="2"> <p><strong>Thomas Cataldo </strong>/ <a title="Linkedin" href="https://www.linkedin.com/in/thomascataldo/"><img src="cid:x-disclaimer-1318405639-0.png@bm-disclaimer" alt="Linkedin" width="15" height="15"></a> <br>Directeur Technique<br>(+33) 6 42 25 91 38</p> </td>
</tr>
<tr>
<td style="vertical-align: top; width: 150.739px;"><img src="cid:x-disclaimer-1318405639-1.png@bm-disclaimer" alt="BlueMind Logo" width="125" height="34"></td>
<td style="width: 399.83px;"> <p style="color: #212121; font-family: Helvetica, sans-serif; font-size: 0.9em;"><strong>BlueMind</strong><br>+33 (0)5 81 91 55 60<br>Hôtel des Télécoms, <br>40 rue du village d'entreprises<br>31670 Labège, France<br><a href="http://www.bluemind.net">www.bluemind.net</a> / <a href="https://blog.bluemind.net/fr/">Blog</a></p> <p style="color: #212121; font-family: Helvetica, sans-serif;"><a title="Linkedin" href="https://www.linkedin.com/company/blue-mind/?originalSubdomain=fr"><img src="cid:x-disclaimer-1318405639-2.png@bm-disclaimer" alt="Linkedin" width="15" height="15"></a> <a title="Facebook" href="https://www.facebook.com/BlueMind.Mail/"><img src="cid:x-disclaimer-1318405639-3.png@bm-disclaimer" alt="Facebook" width="15" height="15"></a> <a title="Twitter" href="https://twitter.com/_bluemind?lang=fr"><img src="cid:x-disclaimer-1318405639-4.png@bm-disclaimer" alt="Twitter" width="15" height="15"></a></p> </td>
</tr>
</tbody>
</table>
<p><a href="https://www.youtube.com/watch?v=WVyRhFgjRMQ&list=PLJj7-LyvGRsI3oP2D8m5cbRghXwD3hBEG&index=2"><em><span style="font-family: helvetica, arial, sans-serif; font-size: 8pt;">Voir l'interview du Président de BlueMind sur la souveraineté numérique et la stratégie de BlueMind</span></em></a></p>
</div>
</body>
</html>