replication together with object storage (cyrus 3.0)

Wed Aug 28 06:04:34 EDT 2019

Hi,

We are working with a replication enabled cyrus 3.0.8.
Our replication endpoint is not another cyrus but I don’t think it matters here. I am working on enabling object storage support with our setup.

Our replication settiings look like this :
sync_log: 1
sync_log_channels: core

core_sync_authname: admin0
core_sync_password: xxxxx
core_sync_realm: repl_realm
core_sync_host: 172.16.167.129
core_sync_port: 2501
core_sync_repeat_interval: 0
core_sync_try_imap: 0

We made a copy of objectstore_dummy.c and simplified it to hardcode the path to /dummy-sds/<message guid> for testing purposes.
The object storage/archive setup looks like this :
object_storage_enabled: 1
archive_enabled: 1
archive_days: 0
archive_maxsize: 0
archive_keepflagged: 0

>From an imap point of view, everything works fine, emails are written through the object storage and are read from there.

# ls -l /dummy-sds/
...
-rw------- 1 cyrus mail 1491 Aug 28 11:35 ae3aaefa50d04042bc369e29e9c819ed600d3a03
-rw------- 1 cyrus mail  863 Aug 28 11:35 f996f4ad4e4c37c3c2cc852214fdeca778d9c43f

The problem occurs when sync_client triggers its replication code.

>From our logs we see:
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: creating sql_db /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/message.db
Aug 28 11:35:23 bm1804 cyrus/imap[116313]: USAGE admin at devenv.blue user: 0.000000 sys: 0.010090
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES devenv.blue!user.admin.Sent devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: Delivered: <ffe06ca2d52c5d64a0d5401f5f23a29a at devenv.blue> to mailbox: devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX devenv.blue!user.admin.Sent
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: IOERROR: Failed to read file /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/a/user/admin/Sent/4.
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: USAGE tom at devenv.blue user: 0.000000 sys: 0.011507
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: IOERROR: Failed to read file /var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/1.
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/imap[33172]: session initialised for admin at devenv.blue
Aug 28 11:35:23 bm1804 cyrus/imap[33172]: login: bm1804.devenv.blue [172.16.167.129] admin at devenv.blue PLAIN User logged in SESSIONID=<cyrus-33172-1566984923-1-5992269715888192037>

>From a protocol point of view we receive (REPL C is what sync_client sends us, REPL S is what we respond):

2019-08-28 09:35:23,310 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000001]: GET MAILBOXES (devenv.blue!user.admin.Sent devenv.blue!user.tom)
2019-08-28 09:35:23,326 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000001]: * MAILBOX %(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME devenv.blue!user.admin.Sent SYNC_CRC 583232180 SYNC_CRC_ANNOT 0 LAST_UID 3 HIGHESTMODSEQ 6 RECENTUID 3 RECENTTIME 1566981904 LAST_APP... [truncated]
2019-08-28 09:35:23,329 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000002]: APPLY RESERVE %(PARTITION bm-master__devenv_blue MBOXNAME (devenv.blue!user.admin.Sent devenv.blue!user.tom) GUID (f996f4ad4e4c37c3c2cc852214fdeca778d9c43f ae3aaefa50d04042bc369e29e9c819ed600d3a03))
2019-08-28 09:35:23,332 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000002]: * MISSING (f996f4ad4e4c37c3c2cc852214fdeca778d9c43f ae3aaefa50d04042bc369e29e9c819ed600d3a03)
OK success
2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000003]: APPLY MESSAGE (NIL)
2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000003]: OK success
2019-08-28 09:35:23,337 [vert.x-eventloop-thread-1] n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000004]: APPLY MAILBOX %(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME devenv.blue!user.admin.Sent SYNC_CRC 3650189183 SYNC_CRC_ANNOT 0 LAST_UID 4 HIGHESTMODSEQ 7 RECENTUID 3 RECENTTIME 1566981904 LAST_APPENDDATE 1566984923 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1566980951 PARTITION bm-master__devenv_blue ACL "admin0	lrswipkxtecdan	E8F1E28F-8778-442B-A581-C6B613CE9555 at devenv.blue	lrswipktecdan	" OPTIONS P ANNOTATIONS (%(ENTRY /specialuse USERID admin at devenv.blue VALUE {5+}{t28.bin})) RECORD (%(UID 4 MODSEQ 7 LAST_UPDATED 1566984923 FLAGS (\Seen) INTERNALDATE 1566984923 SIZE 863 GUID f996f4ad4e4c37c3c2cc852214fdeca778d9c43f)))

The problem seems pretty “obvious” : sync_client APPLY RESERVE for the 2 bodies (from Sent folder + the copy for the recipient inbox), the it tries to read the bodies but I imagine the sync_client code is not object-storage enabled correctly and only tries the local filesystem path instead of asking to the object storage.

It could also be an expected behaviour, as the replica(s) can share the object storage with master and should reply that nothing is missing.

What is your point of view on that ? Could you give me a point to where in the sync_client code I should look to “object storage”-enable it.

Quite un-related but we enjoyed reading your stuff about sync improvements and having multiple sync streams between master & replica. If we can help on that we would be happy to. 

Regards,
Thomas.

Thomas Cataldo
Directeur Technique 

(+33) 6 42 25 91 38

BlueMind
+33 (0)5 81 91 55 60
Hotel des Télécoms, 40 rue du village d'entreprises
31670 Labège, France
www.bluemind.net / https://blog.bluemind.net/fr/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x-disclaimer-1318405639-0.png
Type: image/png
Size: 432 bytes
Desc: x-disclaimer-1318405639-0.png
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x-disclaimer-1318405639-1.png
Type: image/png
Size: 858 bytes
Desc: x-disclaimer-1318405639-1.png
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x-disclaimer-1318405639-2.png
Type: image/png
Size: 432 bytes
Desc: x-disclaimer-1318405639-2.png
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x-disclaimer-1318405639-3.png
Type: image/png
Size: 400 bytes
Desc: x-disclaimer-1318405639-3.png
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x-disclaimer-1318405639-4.png
Type: image/png
Size: 407 bytes
Desc: x-disclaimer-1318405639-4.png
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20190828/2253513e/attachment-0009.png>