backup replication: sync_try_imap

Mon Sep 3 22:50:16 EDT 2018

Hi Philippe,

>From your other email, it looks like you're using the experimental backup system?

Please note that this system is _experimental_ and may change (probably in non-backward compatible ways) in the future.  It's also not really under active development at the moment.  If you don't already have an existing backup system for your Cyrus installation, I would strongly recommend setting something up outside of Cyrus while you experiment with this system.

That being said, if you're experimenting with it, feedback is greatly appreciated! :)

Your backup server needs to be a clean/new Cyrus instance that only runs the backupd service.  It will not have a conventional mailstore, therefore the other Cyrus services (imapd, nntpd, pop3d, etc) will not function -- the backup server's storage is optimised towards long term storage, not interactive (client) use.  If you have already run a normal (non-backupd) replication to the backup server, you'll probably need to wipe it and start again -- the effects of having both a conventional mailstore and a backup store on the same server are untested and unknown. (At the very least, it will be a waste of space: everything will be stored twice!).

On your backup server, you need to run the backupd service (listening on the csync port), not the imapd service.  (If you use the imapd service, you will just have a normal replica, and messages will be deleted in the usual way when the deletion replicates.  Great as a hot spare, useless as a backup!)

On your master server, sync_client will use the specified sync_port if it is set, and speak CSYNC to it.  If sync_port is not set, it will consider sync_try_imap: if the value is "yes", then it will try connecting on port 143 and speaking IMAP.  If this fails, or if the value was "no", then it will connect on the csync port (usually 2005) and speaking CSYNC.  Since your config specifies a "backup_sync_port", the "backup_sync_try_imap" setting is ignored entirely.

The backupd service does not understand IMAP, it speaks only the CSYNC protocol.  So you should have it listen on the csync port.

I think a configuration like this should get you going:

master
> --%snip%--
> # imapd.conf
> sync_log: yes
> sync_log_channels: backup
> backup_sync_host: my.backupserver.local
> backup_sync_try_imap: no
> backup_sync_authname: cyrus
> backup_sync_password: somepassword
> backup_sync_repeat_interval: 1
> --%snip%--

(Notice that I have removed the backup_sync_port, so it will look at the backup_sync_try_imap value. And I have set backup_sync_try_imap to no, so that it won't waste time trying to connect to a service that doesn't exist, and will just connect directly to csync immediately.)

> replica:
> --%snip%--
> # cyrus.conf
> SERVICES {
>   backup cmd="backupd" listen="*:csync"
> }
> --%snip%--

(Notice that I have removed the imapd service, and replaced it with a backupd service that listens on the csync port.)

It's been a while since I looked at the backup system, so I might have forgotten something.  The documentation is accurate, but may be confusing?  Also if you haven't done so recently, I strongly recommend reading the documentation for the all the sync_* options in the imapd.conf(5) man page, they probably contain details that aren't included in the more general setup guide.

Hope this helps! :)

Cheers,

ellie

On Fri, Aug 31, 2018, at 9:18 PM, Philippe wrote:
> Hi all,
> 
> 
> I'm using cyrus-imapd 3.0.8 and I'm trying to set up a rolling backup
> over imap as csync is marked as obsolete.
> 
> According to the documentation I configured the servers this way:
> 
> master:
> --%snip%--
> # imapd.conf
> sync_log: yes
> sync_log_channels: backup
> backup_sync_host: my.backupserver.local
> backup_sync_port: imap
> backup_sync_try_imap: yes
> backup_sync_authname: cyrus
> backup_sync_password: somepassword
> backup_sync_repeat_interval: 1
> --%snip%--
> 
> replica:
> --%snip%--
> # cyrus.conf
> SERVICES {
>   imap cmd="imapd" listen="*:imap"
> }
> --%snip%--
> 
> Testing:
> > $ echo ". logout" | nc my.backupserver.local 143
> > * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR] my.backupserver.local Cyrus IMAP 3.0.8-1.el7 RHEL server ready         
> > * BYE LOGOUT received
> > . OK Completed
> 
> As far as I understood this should be sufficient to make sync_client use
> imap in favor of csync. Unfortunately, sync_client isn't sending valid
> imap commands to the server.
> 
> > $ strace -y -yy -e connect,read,write -s 1024 sync_client -n backup -o -l -v -u user1 |& egrep '(TCP|pipe):'
> > connect(5<TCP:[44275807]>, {sa_family=AF_INET, sin_port=htons(143), sin_addr=inet_addr("192.168.1.253")}, 16) = -1 EINPROGRESS (Operation now in progress)
> > read(5<TCP:[172.31.254.1:44144->192.168.1.253:143]>, "* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR] my.backupserver.local Cyrus IMAP 3.0.8-1.el7 RHEL server ready\r\n", 4096) = 174
> > write(5<TCP:[172.31.254.1:44144->192.168.1.253:143]>, "EXIT\r\n", 6) = 6
> > read(5<TCP:[172.31.254.1:44144->192.168.1.253:143]>, "* BAD Invalid tag\r\n", 4096) = 19
> > write(2<pipe:[44272774]>, "Can not connect to server 'my.backupserver.local'\n", 61Can not connect to server 'my.backupserver.local'
> 
> Now when I change "backup_sync_try_imap: yes" to "sync_try_imap: yes" it
> seems to be working at first but then another connection opens and fails:
> 
> > $ strace -y -yy -e connect,read,write -s 1024 sync_client -n backup -o -l -v -u user1 |& egrep '(TCP|pipe):'
> > connect(5<TCP:[44278756]>, {sa_family=AF_INET, sin_port=htons(143), sin_addr=inet_addr("192.168.1.253")}, 16) = -1 EINPROGRESS (Operation now in progress)
> > read(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR] my.backupserver.local Cyrus IMAP 3.0.8-1.el7 RHEL server ready\r\n", 4096) = 174
> > write(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "A01 AUTHENTICATE DIGEST-MD5\r\n", 29) = 29
> > read(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "+ ...", 4096) = 276
> > write(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "...", 410) = 410
> > read(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "+ cnNwYXV0aD05Y2E1ZDgyNmMyNTZkNTc0ODhlNmFiMDBkMTVjNjY0Mw==\r\n", 4096) = 60
> > write(5<TCP:[172.31.254.1:44420->192.168.1.253:143]>, "...", 36) = 36
> > connect(5<TCP:[44278759]>, {sa_family=AF_INET, sin_port=htons(143), sin_addr=inet_addr("192.168.1.253")}, 16) = -1 EINPROGRESS (Operation now in progress)
> > read(5<TCP:[172.31.254.1:44424->192.168.1.253:143]>, "* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR] my.backupserver.local Cyrus IMAP 3.0.8-1.el7 RHEL server ready\r\n", 4096) = 174
> > write(5<TCP:[172.31.254.1:44424->192.168.1.253:143]>, "EXIT\r\n", 6) = 6
> > read(5<TCP:[172.31.254.1:44424->192.168.1.253:143]>, "* BAD Invalid tag\r\n", 4096) = 19
> > write(2<pipe:[44306013]>, "Can not connect to server 'my.backupserver.local'\n", 61Can not connect to server 'my.backupserver.local'
> 
> Am I missing something?
> 
> 
> Best,
> 
> Philippe