From debbiep at polyfoam.com.au Wed Apr 1 01:09:04 2020 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Wed, 1 Apr 2020 16:09:04 +1100 Subject: Sendmail+Cyrus IMAP "Data format error" and bounces In-Reply-To: References: <9ef2e539-b264-d11b-fda2-5b42e92f67b3@netfence.it> <202003171631.32524.a_s_y@sama.ru> <2f1aef32-4b1b-f37b-1a27-22cf9a6e067f@netfence.it> Message-ID: <7ca270a5-0052-468e-e45d-807b909a68dd@polyfoam.com.au> On 2020-03-28 04.59, Andrea Venturoli wrote: > Any idea for a similar tool which can use UTF8SMTPS, so I can repeat > that tests? I would be going low-tech and using telnet or netcat or socat to converse directly with your mail server's port 25.? Craft the client's side of the conversation in another window and copy-paste lines directly to the SMTP service.? This way you can tweak the exact conversation and hopefully trigger the behaviour you are intermittently seeing. From olaf at navi.pl Sat Apr 4 09:45:32 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Sat, 4 Apr 2020 15:45:32 +0200 Subject: Replication - current status and how to do failover Message-ID: Hello, 1. Is currently master-master replication possible (maybe 3.2) Is it OK to sync them two-way? If yes - how to set up such config? 2. If master-master is impossible, is there any guide how to setup failover from master to slave and possibly back? If split-brain happens - is there an easy recovery from such state? Best regards, Olaf From brong at fastmail.fm Tue Apr 7 23:53:54 2020 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 08 Apr 2020 13:53:54 +1000 Subject: Replication - current status and how to do failover In-Reply-To: References: Message-ID: <5ce10f7e-0d6d-43fc-81e0-9c394938a0ef@dogfood.fastmail.com> On Sun, Apr 5, 2020, at 00:45, Olaf Fr?czyk wrote: > Hello, > > 1. Is currently master-master replication possible (maybe 3.2) Is it OK > to sync them two-way? No, not really. It'll mostly be fine, but it doesn't (yet) handle folder create/rename/delete safely. > If yes - how to set up such config? > > 2. If master-master is impossible, is there any guide how to setup > failover from master to slave and possibly back? If split-brain happens > - is there an easy recovery from such state? The way we do it at Fastmail is with an nginx proxy in front which knows which one is the master. For a clean shutdown, we shut down the master, then run sync_client -r -f with the log file (if anything was unreplicated) to make sure it's up to date, then shut down both and bring them up with the config pointing the replication the other way. For a case where the master crashed hard, we switch the replica to be master by changing the config (with a restart again) then bring the old master back up and for sync_client everything again like above to switch back, so all new changes from the regular replica are back on the regular master. Then we bring up the regular master as master again, and run sync_client -A from there to replicate all remaining changes. That mostly works. The plan in 3.4+ is to use the mailbox tombstone records to get the create/rename/delete to the same level of split-brain safety as the UIDs inside the mailbox have. Cheers, Bron. -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- An HTML attachment was scrubbed... URL: From a_s_y at sama.ru Fri Apr 10 04:39:09 2020 From: a_s_y at sama.ru (Sergey) Date: Fri, 10 Apr 2020 12:39:09 +0400 Subject: Sendmail+Cyrus IMAP "Data format error" and bounces In-Reply-To: References: <9ef2e539-b264-d11b-fda2-5b42e92f67b3@netfence.it> <2f1aef32-4b1b-f37b-1a27-22cf9a6e067f@netfence.it> Message-ID: <202004101239.09862.a_s_y@sama.ru> On Friday 27 March 2020, you wrote: > With "reject8bit: yes", sendmail refuses the message with "Message > contains non-ASCII characters in headers". (Still don't know why it > doesn't refuse it upfront, but bounces it!). This is LMTP error between Sendmail and Cyrus. The message already received by Sendmail and Sendmail must send a bounce at this case. I use "reject8bit: 0" and "munge8bit: 0" in imapd.conf. > With "reject8bit: no", I get the message and the subject is translated > to "ProvaXX", as I would expect, since rfc2047_utf8 is, by default, off. > > Strangely, if I enable rfc2047_utf8, I still get "ProvaXX" as a subject. https://www.cyrusimap.org/imap/reference/manpages/configs/imapd.conf.html | munge8bit: 1 | | If enabled, lmtpd munges messages with 8-bit characters in the headers. | The 8-bit characters are changed to `X?. If reject8bit is enabled, | setting munge8bit has no effect. -- Regards, Sergey From stanimir at audriga.com Wed Apr 15 01:34:58 2020 From: stanimir at audriga.com (stanimir at audriga.com) Date: Wed, 15 Apr 2020 07:34:58 +0200 Subject: Problems during local JMAP setup regarding HTTP 405 response Message-ID: Hello everyone, I'm currently trying to test a JMAP client library against a locally running Cyrus instance. The issue which I'm experiencing is that when trying to make a JMAP request like the one described in the docs here: https://www.cyrusimap.org/dev/imap/developer/jmap.html, I'm getting a "405 Method Not Allowed" response. My setup is: - OS: Debian Bullseye - Cyrus version: 3.2.0-beta3-Debian-3.2.0~beta3-1 (I got the Cyrus as a Debian package and managed to get it running) Here are my .conf files: === cyrus.conf === START { recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r" delprune cmd="/usr/sbin/cyrus expire -E 3" tlsprune cmd="/usr/sbin/cyrus tls_prune" } SERVICES { imap cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100 #imaps cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100 pop3 cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50 #pop3s cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50 nntp cmd="nntpd -U 30" listen="nntp" prefork=0 maxchild=100 #nntps cmd="nntpd -s -U 30" listen="nntps" prefork=0 maxchild=100 #http cmd="httpd -U 30" listen="8008" prefork=0 maxchild=100 #https cmd="httpd -s -U 30" listen="8443" prefork=0 maxchild=100 http cmd="httpd" listen="http" prefork=0 https cmd="httpd -s" listen="https" prefork=0 lmtpunix cmd="lmtpd" listen="/run/cyrus/socket/lmtp" prefork=0 maxchild=20 sieve cmd="timsieved" listen="localhost:sieve" prefork=0 maxchild=100 notify cmd="notifyd" listen="/run/cyrus/socket/notify" proto="udp" prefork=1 } EVENTS { checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30 delprune cmd="/usr/sbin/cyrus expire -E 3" at=0401 tlsprune cmd="/usr/sbin/cyrus tls_prune" at=0401 deleteprune cmd="/usr/sbin/cyrus expire -E 4 -D 28" at=0430 expungeprune cmd="/usr/sbin/cyrus expire -E 4 -X 28" at=0445 } DAEMON { squatter cmd="squatter -R" } === imapd.conf === configdirectory: /var/lib/cyrus proc_path: /run/cyrus/proc mboxname_lockpath: /run/cyrus/lock #defaultpartition: default partition-default: /var/spool/cyrus/mail # News setup partition-news: /var/spool/cyrus/news newsspool: /var/spool/news altnamespace: no unixhierarchysep: 1 lmtp_downcase_rcpt: yes admins: cyrus imapuser allowanonymouslogin: no popminpoll: 1 autocreate_quota: 0 umask: 077 sieveusehomedir: false sievedir: /var/spool/sieve httpmodules: caldav carddav jmap hashimapspool: true allowplaintext: yes sasl_pwcheck_method: auxprop sasl_auto_transition: no tls_client_ca_dir: /etc/ssl/certs tls_session_timeout: 1440 lmtpsocket: /run/cyrus/socket/lmtp idlesocket: /run/cyrus/socket/idle notifysocket: /run/cyrus/socket/notify syslog_prefix: cyrus sync_log: on sync_log_channels: squatter search_engine: xapian search_index_headers: no search_batchsize: 8192 defaultpartition: base defaultsearchtier: t1 partition-base: /var/cyrus/spool t1searchpartition-base: /var/cyrus/search conversations: 1 conversations_db: twoskip calendarprefix: #calendars I also ran an OPTIONS request for all paths (with the * operator) on port 80 for localhost via telnet, which showed that pretty much all HTTP methods are possible (including DAV). However, when running the OPTIONS request for the /jmap route, only OPTIONS, GET and HEAD are allowed. I would be very happy if somebody could offer some insights on this issue. Best regards, Stanimir Bozhilov From murch at fastmail.com Wed Apr 15 06:40:25 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 15 Apr 2020 06:40:25 -0400 Subject: Problems during local JMAP setup regarding HTTP 405 response In-Reply-To: References: Message-ID: <91416b07-1e1c-1b37-8081-60f49745cd69@fastmail.com> Are your requests against /jmap or /jmap/ (note that the trailing / is required)? On 4/15/20 1:34 AM, stanimir at audriga.com wrote: > Hello everyone, > > I'm currently trying to test a JMAP client library against a locally > running Cyrus instance. > The issue which I'm experiencing is that when trying to make a JMAP > request like the one described in the docs here: > https://www.cyrusimap.org/dev/imap/developer/jmap.html, I'm getting a > "405 Method Not Allowed" response. > > My setup is: > ?- OS: Debian Bullseye > ?- Cyrus version: 3.2.0-beta3-Debian-3.2.0~beta3-1 (I got the Cyrus as > a Debian package and managed to get it running) > > Here are my .conf files: > > === cyrus.conf === > START { > ??? recover??????? cmd="/usr/sbin/cyrus ctl_cyrusdb -r" > ??? delprune??? cmd="/usr/sbin/cyrus expire -E 3" > ??? tlsprune??? cmd="/usr/sbin/cyrus tls_prune" > } > > SERVICES { > ??? imap??????? cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100 > ??? #imaps??????? cmd="imapd -s -U 30" listen="imaps" prefork=0 > maxchild=100 > ??? pop3??????? cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50 > ??? #pop3s??????? cmd="pop3d -s -U 30" listen="pop3s" prefork=0 > maxchild=50 > ??? nntp??????? cmd="nntpd -U 30" listen="nntp" prefork=0 maxchild=100 > ??? #nntps??????? cmd="nntpd -s -U 30" listen="nntps" prefork=0 > maxchild=100 > ??? #http??????? cmd="httpd -U 30" listen="8008" prefork=0 maxchild=100 > ??? #https??????? cmd="httpd -s -U 30" listen="8443" prefork=0 > maxchild=100 > ??? http??????? cmd="httpd" listen="http" prefork=0 > ??? https??????? cmd="httpd -s" listen="https" prefork=0 > > ??? lmtpunix??? cmd="lmtpd" listen="/run/cyrus/socket/lmtp" prefork=0 > maxchild=20 > ??? sieve??????? cmd="timsieved" listen="localhost:sieve" prefork=0 > maxchild=100 > ??? notify??????? cmd="notifyd" listen="/run/cyrus/socket/notify" > proto="udp" prefork=1 > } > > EVENTS { > ??? checkpoint??? cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30 > ??? delprune??? cmd="/usr/sbin/cyrus expire -E 3" at=0401 > ??? tlsprune??? cmd="/usr/sbin/cyrus tls_prune" at=0401 > ??? deleteprune??? cmd="/usr/sbin/cyrus expire -E 4 -D 28" at=0430 > ??? expungeprune??? cmd="/usr/sbin/cyrus expire -E 4 -X 28" at=0445 > } > > DAEMON { > ??? squatter cmd="squatter -R" > } > > > === imapd.conf === > configdirectory: /var/lib/cyrus > > proc_path: /run/cyrus/proc > mboxname_lockpath: /run/cyrus/lock > > #defaultpartition: default > partition-default: /var/spool/cyrus/mail > > # News setup > partition-news: /var/spool/cyrus/news > newsspool: /var/spool/news > > altnamespace: no > > unixhierarchysep: 1 > > lmtp_downcase_rcpt: yes > > admins: cyrus imapuser > > allowanonymouslogin: no > > popminpoll: 1 > > autocreate_quota: 0 > > umask: 077 > > sieveusehomedir: false > > sievedir: /var/spool/sieve > > httpmodules: caldav carddav jmap > > hashimapspool: true > > allowplaintext: yes > > sasl_pwcheck_method: auxprop > > sasl_auto_transition: no > > tls_client_ca_dir: /etc/ssl/certs > > tls_session_timeout: 1440 > > lmtpsocket: /run/cyrus/socket/lmtp > > idlesocket: /run/cyrus/socket/idle > > notifysocket: /run/cyrus/socket/notify > > syslog_prefix: cyrus > > sync_log: on > sync_log_channels: squatter > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultpartition: base > defaultsearchtier: t1 > partition-base: /var/cyrus/spool > t1searchpartition-base: /var/cyrus/search > > conversations: 1 > conversations_db: twoskip > > > calendarprefix: #calendars > > > > I also ran an OPTIONS request for all paths (with the * operator) on > port 80 for localhost via telnet, which showed that pretty much all > HTTP methods are possible (including DAV). However, when running the > OPTIONS request for the /jmap route, only OPTIONS, GET and HEAD are > allowed. > > I would be very happy if somebody could offer some insights on this > issue. > > > Best regards, > > Stanimir Bozhilov > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Ken Murchison Cyrus Development Team Fastmail US LLC From ellie at fastmail.com Mon Apr 20 03:16:48 2020 From: ellie at fastmail.com (ellie timoney) Date: Mon, 20 Apr 2020 17:16:48 +1000 Subject: Cyrus IMAP 3.2.0-rc1 released Message-ID: <855e57e7-cf48-454e-95bc-6b3ceb660c0c@www.fastmail.com> The Cyrus team is proud to announce the first release candidate from the new Cyrus IMAP 3.2 series: 3.2.0-rc1 While 3.2 is still in pre-release, the main https://www.cyrusimap.org/ website will continue to be focused on the stable 3.0 series. The 3.2 website is available at https://www.cyrusimap.org/3.2/ Download URLs: https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-rc1/cyrus-imapd-3.2.0-rc1.tar.gz https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-rc1/cyrus-imapd-3.2.0-rc1.tar.gz.sig Please consult the release notes and upgrade documentation before upgrading to 3.2: https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-rc1.html https://www.cyrusimap.org/3.2/imap/download/upgrade.html And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation. On behalf of the Cyrus team, Kind regards, ellie timoney From olaf at navi.pl Mon Apr 20 10:11:08 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Mon, 20 Apr 2020 16:11:08 +0200 Subject: Replication failed 3.0.5 -> 3.0.13 Message-ID: Hi, I'm running 3.0.5. I want to migrate to a new machine. I set up cyrus-imapd 3.0.13. The replication started but it didn't transfer all mails. The store isn't big 44GB, transferred was about 24 GB. In the logs I see: Apr 20 14:54:03 ifs sync_client[24239]: couldn't authenticate to backend server: authentication failure Apr 20 14:54:13 ifs sync_client[24239]: connect(skink1.navi.pl) failed: Connection timed out Apr 20 14:58:13 ifs sync_client[24239]: auditlog: proxy sessionid= remote= Apr 20 15:12:41 ifs sync_client[24239]: sync_client RESTART succeeded Apr 20 15:12:42 ifs sync_client[24239]: auditlog: proxy sessionid= remote= Apr 20 15:12:46 ifs sync_client[24239]: inefficient replication (39865 > 48) navi.pl!user.info Apr 20 15:15:31 ifs sync_client[24239]: inefficient replication (32058 > 56) navi.pl!user.olaf Apr 20 15:18:50 ifs sync_client[24239]: inefficient replication (15216 > 13867) navi.pl!user.ania Apr 20 15:19:18 ifs sync_client[24239]: inefficient replication (56949 > 432) navi.pl!user.piotr Apr 20 15:25:11 ifs sync_client[24239]: sync_client RESTART succeeded Apr 20 15:25:12 ifs sync_client[24239]: auditlog: proxy sessionid= remote= Apr 20 15:29:09 ifs sync_client[24239]: IOERROR: zero length response to MAILBOXES (idle for too long) Apr 20 15:29:09 ifs sync_client[24239]: IOERROR: zero length response to RESTART (idle for too long) Apr 20 15:29:09 ifs sync_client[24239]: Error in do_sync(): bailing out! Bad protocol Apr 20 15:29:09 ifs sync_client[24239]: Processing sync log file /var/lib/imap/sync/log-run failed: Bad protocol I also executed thereafter sync client manually: /usr/local/cyrus-3.0.5/sbin/sync_client -r but nothing changed, in the log appeared: Apr 20 15:37:23 ifs sync_client[27304]: Reprocessing sync log file /var/lib/imap/sync/log-run The directory /var/lib/imap/sync/ on master is empty. The directory /var/spool/imap/sync. on replica is also empty What can I try to get it woring? Is there a serious problem with replication in 3.0.5? I wanted to avoid upgrade on the old machine. Best regrads, Olaf Fr?czyk From olaf at navi.pl Mon Apr 20 22:49:50 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 04:49:50 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: References: Message-ID: Hi, I upgraded to 3.0.13 but it didn't help. This time it copied about 18GB in the logs I still see: 1 - inefficient replication 2 - IOERROR: zero length response to MAILBOXES (idle for too long) IOERROR: zero length response to RESTART (idle for too long) Error in do_sync(): bailing out! Bad protocol But I have no idea what can I do next and why it fails Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length response to MAILBOXES (idle for too long) Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length response to RESTART (idle for too long) Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing out! Bad protocol Apr 21 02:24:46 ifs sync_client[12656]: Processing sync log file /var/lib/imap/sync/log-run failed: Bad protocol Apr 21 03:46:10 ifs sync_client[1353]: auditlog: proxy sessionid= remote= Apr 21 03:53:15 ifs sync_client[1353]: inefficient replication (32058 > 56) navi.pl!user.olaf Apr 21 03:59:41 ifs sync_client[1353]: sync_client RESTART succeeded Apr 21 03:59:41 ifs sync_client[1353]: auditlog: proxy sessionid= remote= Apr 21 04:01:49 ifs sync_client[1353]: inefficient replication (56949 > 432) navi.pl!user.piotr Apr 21 04:10:38 ifs sync_client[1353]: sync_client RESTART succeeded Apr 21 04:10:38 ifs sync_client[1353]: auditlog: proxy sessionid= remote= Apr 21 04:20:39 ifs sync_client[1353]: sync_client RESTART succeeded Apr 21 04:20:39 ifs sync_client[1353]: auditlog: proxy sessionid= remote= Apr 21 04:30:34 ifs sync_client[1353]: IOERROR: zero length response to MAILBOXES (idle for too long) Apr 21 04:30:34 ifs sync_client[1353]: IOERROR: zero length response to RESTART (idle for too long) Apr 21 04:30:34 ifs sync_client[1353]: Error in do_sync(): bailing out! Bad protocol Regards, Olaf On 2020-04-20 16:11, Olaf Fr?czyk wrote: > Hi, > > I'm running 3.0.5. > > I want to migrate to a new machine. I set up cyrus-imapd 3.0.13. > > The replication started but it didn't transfer all mails. > > The store isn't big 44GB, transferred was about 24 GB. > > In the logs I see: > > Apr 20 14:54:03 ifs sync_client[24239]: couldn't authenticate to > backend server: authentication failure > Apr 20 14:54:13 ifs sync_client[24239]: connect(skink1.navi.pl) > failed: Connection timed out > Apr 20 14:58:13 ifs sync_client[24239]: auditlog: proxy > sessionid= > remote= > Apr 20 15:12:41 ifs sync_client[24239]: sync_client RESTART succeeded > Apr 20 15:12:42 ifs sync_client[24239]: auditlog: proxy > sessionid= > remote= > Apr 20 15:12:46 ifs sync_client[24239]: inefficient replication (39865 > > 48) navi.pl!user.info > Apr 20 15:15:31 ifs sync_client[24239]: inefficient replication (32058 > > 56) navi.pl!user.olaf > Apr 20 15:18:50 ifs sync_client[24239]: inefficient replication (15216 > > 13867) navi.pl!user.ania > Apr 20 15:19:18 ifs sync_client[24239]: inefficient replication (56949 > > 432) navi.pl!user.piotr > Apr 20 15:25:11 ifs sync_client[24239]: sync_client RESTART succeeded > Apr 20 15:25:12 ifs sync_client[24239]: auditlog: proxy > sessionid= > remote= > Apr 20 15:29:09 ifs sync_client[24239]: IOERROR: zero length response > to MAILBOXES (idle for too long) > Apr 20 15:29:09 ifs sync_client[24239]: IOERROR: zero length response > to RESTART (idle for too long) > Apr 20 15:29:09 ifs sync_client[24239]: Error in do_sync(): bailing > out! Bad protocol > Apr 20 15:29:09 ifs sync_client[24239]: Processing sync log file > /var/lib/imap/sync/log-run failed: Bad protocol > > I also executed thereafter sync client manually: > > /usr/local/cyrus-3.0.5/sbin/sync_client -r > > but nothing changed, in the log appeared: > > Apr 20 15:37:23 ifs sync_client[27304]: Reprocessing sync log file > /var/lib/imap/sync/log-run > > The directory /var/lib/imap/sync/ on master is empty. > > The directory /var/spool/imap/sync. on replica is also empty > > What can I try to get it woring? Is there a serious problem with > replication in 3.0.5? I wanted to avoid upgrade on the old machine. > > Best regrads, > > Olaf Fr?czyk > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Tue Apr 21 02:47:18 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 21 Apr 2020 08:47:18 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: References: Message-ID: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> Hi Olaf Quoting Olaf Fr?czyk : > Hi, > > I upgraded to 3.0.13 but it didn't help. > > This time it copied about 18GB > > in the logs I still see: > > 1 - inefficient replication > > 2 - IOERROR: zero length response to MAILBOXES (idle for too long) > IOERROR: zero length response to RESTART (idle for too long) > Error in do_sync(): bailing out! Bad protocol > > But I have no idea what can I do next and why it fails > > Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length > response to MAILBOXES (idle for too long) > Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length > response to RESTART (idle for too long) > Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing > out! Bad protocol do you see any errors on the syncserver side. The error look like the sync_client is waiting for a reply of the server. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From olaf at navi.pl Tue Apr 21 05:07:33 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 11:07:33 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> Message-ID: <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> Hi, When I run sync_client -r on the master I see the following on the replica: Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock navi.pl!user.olaf for 1.7 seconds Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock navi.pl!user.piotr for 2.0 seconds Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock navi.pl!user.olaf for 2.9 seconds Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock navi.pl!user.piotr for 3.0 seconds The mailboxes have several GB in Inbox folder and several GB in subfolders. The inbox folders have about 20,000 messages, the subfolders upto 15,000 Could it cause problems? Maybe I should move the sync_client from START section to SERVICES, it seems that it is not automatically restarted On 2020-04-21 08:47, Michael Menge wrote: > Hi Olaf > > > Quoting Olaf Fr?czyk : > >> Hi, >> >> I upgraded to 3.0.13 but it didn't help. >> >> This time it copied about 18GB >> >> in the logs I still see: >> >> 1 - inefficient replication >> >> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >> IOERROR: zero length response to RESTART (idle for too long) >> Error in do_sync(): bailing out! Bad protocol >> >> But I have no idea what can I do next and why it fails >> >> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length response >> to MAILBOXES (idle for too long) >> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length response >> to RESTART (idle for too long) >> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing >> out! Bad protocol > > do you see any errors on the syncserver side. The error look like the > sync_client is waiting for a reply of the server. > > > > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From olaf at navi.pl Tue Apr 21 05:18:39 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 11:18:39 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> Message-ID: I also found out that when I see on master: Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length response to MAILBOXES (idle for too long) Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length response to RESTART (idle for too long) Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): bailing out! Bad protocol Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file /var/lib/imap/sync/log-run failed: Bad protocol I also get on the replica: Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, closing connection But I also see, that before it happens, there is no activity both on the replica and on the master for some time. So maybe the imap server process is not recovering correctly in the longlock situation? On 2020-04-21 11:07, Olaf Fr?czyk wrote: > Hi, > > When I run sync_client -r on the master I see the following on the > replica: > > Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock navi.pl!user.olaf > for 1.7 seconds > Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock > navi.pl!user.piotr for 2.0 seconds > Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock navi.pl!user.olaf > for 2.9 seconds > Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock > navi.pl!user.piotr for 3.0 seconds > > The mailboxes have several GB in Inbox folder and several GB in > subfolders. The inbox folders have about 20,000 messages, the > subfolders upto 15,000 > > Could it cause problems? > > Maybe I should move the sync_client from START section to SERVICES, it > seems that it is not automatically restarted > > On 2020-04-21 08:47, Michael Menge wrote: >> Hi Olaf >> >> >> Quoting Olaf Fr?czyk : >> >>> Hi, >>> >>> I upgraded to 3.0.13 but it didn't help. >>> >>> This time it copied about 18GB >>> >>> in the logs I still see: >>> >>> 1 - inefficient replication >>> >>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>> IOERROR: zero length response to RESTART (idle for too long) >>> Error in do_sync(): bailing out! Bad protocol >>> >>> But I have no idea what can I do next and why it fails >>> >>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>> response to MAILBOXES (idle for too long) >>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>> response to RESTART (idle for too long) >>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing >>> out! Bad protocol >> >> do you see any errors on the syncserver side. The error look like the >> sync_client is waiting for a reply of the server. >> >> >> >> >> >> -------------------------------------------------------------------------------- >> >> M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 >> Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 >> Zentrum f?r Datenverarbeitung????????? mail: >> michael.menge at zdv.uni-tuebingen.de >> W?chterstra?e 76 >> 72074 T?bingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From olaf at navi.pl Tue Apr 21 06:58:56 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 12:58:56 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> Message-ID: The current situation is: 1. Replica: stopped and started the replica no activity on replica - iotop and top show nothing the only messages on replica is incoming connection from master 2. Master: when I run sync_client -r I still get: Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file /var/lib/imap/sync/log-run Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length response to MAILBOXES (idle for too long) Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length response to RESTART (idle for too long) Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): bailing out! Bad protocol Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file /var/lib/imap/sync/log-run failed: Bad protocol 3. There is 27 GB of about 45 GB replicated and there is no further progress 4. How to find out why the replica doesn't respond? On 2020-04-21 11:18, Olaf Fr?czyk wrote: > I also found out that when I see on master: > > Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length response > to MAILBOXES (idle for too long) > Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length response > to RESTART (idle for too long) > Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): bailing > out! Bad protocol > Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file > /var/lib/imap/sync/log-run failed: Bad protocol > > I also get on the replica: > > Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, closing > connection > > But I also see, that before it happens, there is no activity both on > the replica and on the master for some time. > > So maybe the imap server process is not recovering correctly in the > longlock situation? > > On 2020-04-21 11:07, Olaf Fr?czyk wrote: >> Hi, >> >> When I run sync_client -r on the master I see the following on the >> replica: >> >> Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock >> navi.pl!user.olaf for 1.7 seconds >> Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock >> navi.pl!user.piotr for 2.0 seconds >> Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock >> navi.pl!user.olaf for 2.9 seconds >> Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock >> navi.pl!user.piotr for 3.0 seconds >> >> The mailboxes have several GB in Inbox folder and several GB in >> subfolders. The inbox folders have about 20,000 messages, the >> subfolders upto 15,000 >> >> Could it cause problems? >> >> Maybe I should move the sync_client from START section to SERVICES, >> it seems that it is not automatically restarted >> >> On 2020-04-21 08:47, Michael Menge wrote: >>> Hi Olaf >>> >>> >>> Quoting Olaf Fr?czyk : >>> >>>> Hi, >>>> >>>> I upgraded to 3.0.13 but it didn't help. >>>> >>>> This time it copied about 18GB >>>> >>>> in the logs I still see: >>>> >>>> 1 - inefficient replication >>>> >>>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>>> IOERROR: zero length response to RESTART (idle for too long) >>>> Error in do_sync(): bailing out! Bad protocol >>>> >>>> But I have no idea what can I do next and why it fails >>>> >>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>> response to MAILBOXES (idle for too long) >>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>> response to RESTART (idle for too long) >>>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing >>>> out! Bad protocol >>> >>> do you see any errors on the syncserver side. The error look like the >>> sync_client is waiting for a reply of the server. >>> >>> >>> >>> >>> >>> -------------------------------------------------------------------------------- >>> >>> M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 >>> Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 >>> Zentrum f?r Datenverarbeitung????????? mail: >>> michael.menge at zdv.uni-tuebingen.de >>> W?chterstra?e 76 >>> 72074 T?bingen >>> >>> ---- >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Tue Apr 21 08:13:05 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 21 Apr 2020 14:13:05 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> Message-ID: <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> Hi, Quoting Olaf Fr?czyk : > The current situation is: > > 1. Replica: > > stopped and started the replica > > no activity on replica - iotop and top show nothing > > the only messages on replica is incoming connection from master > > 2. Master: > > when I run sync_client -r I still get: > > Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file > /var/lib/imap/sync/log-run > Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length > response to MAILBOXES (idle for too long) > Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length > response to RESTART (idle for too long) > Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): bailing > out! Bad protocol > Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file > /var/lib/imap/sync/log-run failed: Bad protocol > > 3. There is 27 GB of about 45 GB replicated and there is no further progress > > 4. How to find out why the replica doesn't respond? > You can enable telemetry logging on the replica by creating a folder /var/lib/imap/log/ where is the value of the sync_authname. If you give cyrus write permissions for this folder it will create log-files for each process and what it received and send from/to the sync_client with timestamps. Also you can try to use strace on the syncserver process to figure out which files it is accessing > On 2020-04-21 11:18, Olaf Fr?czyk wrote: >> I also found out that when I see on master: >> >> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >> response to MAILBOXES (idle for too long) >> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >> response to RESTART (idle for too long) >> Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): bailing >> out! Bad protocol >> Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file >> /var/lib/imap/sync/log-run failed: Bad protocol >> >> I also get on the replica: >> >> Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, >> closing connection >> >> But I also see, that before it happens, there is no activity both >> on the replica and on the master for some time. >> >> So maybe the imap server process is not recovering correctly in the >> longlock situation? >> >> On 2020-04-21 11:07, Olaf Fr?czyk wrote: >>> Hi, >>> >>> When I run sync_client -r on the master I see the following on the replica: >>> >>> Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock >>> navi.pl!user.olaf for 1.7 seconds >>> Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock >>> navi.pl!user.piotr for 2.0 seconds >>> Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock >>> navi.pl!user.olaf for 2.9 seconds >>> Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock >>> navi.pl!user.piotr for 3.0 seconds >>> >>> The mailboxes have several GB in Inbox folder and several GB in >>> subfolders. The inbox folders have about 20,000 messages, the >>> subfolders upto 15,000 >>> >>> Could it cause problems? the longlock is normaly not the problem. While on process has the lock no other process can write to the mailbox, but on the replica there normaly is no other process that should access the mailbox >>> >>> Maybe I should move the sync_client from START section to >>> SERVICES, it seems that it is not automatically restarted >>> I havend tried starting sync_client in the service section. I start the sync_client via systemd. >>> On 2020-04-21 08:47, Michael Menge wrote: >>>> Hi Olaf >>>> >>>> >>>> Quoting Olaf Fr?czyk : >>>> >>>>> Hi, >>>>> >>>>> I upgraded to 3.0.13 but it didn't help. >>>>> >>>>> This time it copied about 18GB >>>>> >>>>> in the logs I still see: >>>>> >>>>> 1 - inefficient replication >>>>> >>>>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>>>> IOERROR: zero length response to RESTART (idle for too long) >>>>> Error in do_sync(): bailing out! Bad protocol >>>>> >>>>> But I have no idea what can I do next and why it fails >>>>> >>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>> response to MAILBOXES (idle for too long) >>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>> response to RESTART (idle for too long) >>>>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): >>>>> bailing out! Bad protocol >>>> >>>> do you see any errors on the syncserver side. The error look like the >>>> sync_client is waiting for a reply of the server. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From olaf at navi.pl Tue Apr 21 08:30:53 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 14:30:53 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> Message-ID: Thank you for the telemetry hint :) I don't use the syncserver - the replication is done via IMAP port on the replica side. I have no idea how to have strace spawned by cyrus master process. When I attach later to imapd using strace -p I'm afraid some info already will be lost. The syncserver is marked as deprecated in the docs, so I went with the more modern option. Maybe here is the problem ;) The funny thing is that from time to time the replication progresses a little. I don't like non-repetitive behavior ;) Thanks again for the hints. On 2020-04-21 14:13, Michael Menge wrote: > Hi, > > Quoting Olaf Fr?czyk : > >> The current situation is: >> >> 1. Replica: >> >> stopped and started the replica >> >> no activity on replica - iotop and top show nothing >> >> the only messages on replica is incoming connection from master >> >> 2. Master: >> >> when I run sync_client -r I still get: >> >> Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file >> /var/lib/imap/sync/log-run >> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length response >> to MAILBOXES (idle for too long) >> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length response >> to RESTART (idle for too long) >> Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): bailing >> out! Bad protocol >> Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file >> /var/lib/imap/sync/log-run failed: Bad protocol >> >> 3. There is 27 GB of about 45 GB replicated and there is no further >> progress >> >> 4. How to find out why the replica doesn't respond? >> > > You can enable telemetry logging on the replica by creating a folder > /var/lib/imap/log/ > where is the value of the? sync_authname. If you give > cyrus write permissions for this folder > it will create log-files for each process and what it received and > send from/to the sync_client with timestamps. > > Also you can try to use strace on the syncserver process to figure out > which files it is accessing > > >> On 2020-04-21 11:18, Olaf Fr?czyk wrote: >>> I also found out that when I see on master: >>> >>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>> response to MAILBOXES (idle for too long) >>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>> response to RESTART (idle for too long) >>> Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): bailing >>> out! Bad protocol >>> Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file >>> /var/lib/imap/sync/log-run failed: Bad protocol >>> >>> I also get on the replica: >>> >>> Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, closing >>> connection >>> >>> But I also see, that before it happens, there is no activity both on >>> the replica and on the master for some time. >>> >>> So maybe the imap server process is not recovering correctly in the >>> longlock situation? >>> >>> On 2020-04-21 11:07, Olaf Fr?czyk wrote: >>>> Hi, >>>> >>>> When I run sync_client -r on the master I see the following on the >>>> replica: >>>> >>>> Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock >>>> navi.pl!user.olaf for 1.7 seconds >>>> Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock >>>> navi.pl!user.piotr for 2.0 seconds >>>> Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock >>>> navi.pl!user.olaf for 2.9 seconds >>>> Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock >>>> navi.pl!user.piotr for 3.0 seconds >>>> >>>> The mailboxes have several GB in Inbox folder and several GB in >>>> subfolders. The inbox folders have about 20,000 messages, the >>>> subfolders upto 15,000 >>>> >>>> Could it cause problems? > > the longlock is normaly not the problem. While on process has the lock > no other process can write to the mailbox, but on the replica there > normaly > is no other process that should access the mailbox > >>>> >>>> Maybe I should move the sync_client from START section to SERVICES, >>>> it seems that it is not automatically restarted >>>> > I havend tried starting sync_client in the service section. > I start the sync_client via systemd. > >>>> On 2020-04-21 08:47, Michael Menge wrote: >>>>> Hi Olaf >>>>> >>>>> >>>>> Quoting Olaf Fr?czyk : >>>>> >>>>>> Hi, >>>>>> >>>>>> I upgraded to 3.0.13 but it didn't help. >>>>>> >>>>>> This time it copied about 18GB >>>>>> >>>>>> in the logs I still see: >>>>>> >>>>>> 1 - inefficient replication >>>>>> >>>>>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>>>>> IOERROR: zero length response to RESTART (idle for too long) >>>>>> Error in do_sync(): bailing out! Bad protocol >>>>>> >>>>>> But I have no idea what can I do next and why it fails >>>>>> >>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>> response to MAILBOXES (idle for too long) >>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>> response to RESTART (idle for too long) >>>>>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): >>>>>> bailing out! Bad protocol >>>>> >>>>> do you see any errors on the syncserver side. The error look like the >>>>> sync_client is waiting for a reply of the server. > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Tue Apr 21 09:00:56 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 21 Apr 2020 15:00:56 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> Message-ID: <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> Quoting Olaf Fr?czyk : > Thank you for the telemetry hint :) > > I don't use the syncserver - the replication is done via IMAP port > on the replica side. I have no idea how to have strace spawned by > cyrus master process. When I attach later to imapd using strace -p > I'm afraid some info already will be lost. > You can try "/usr/bin/strace /imapd " as cmd or you can use prefork=1 in the service to prefok one imap process and connect to this one before you start the sync_client. > The syncserver is marked as deprecated in the docs, so I went with > the more modern option. Maybe here is the problem ;) > i am using syncserver since cyrus-imapd 2.3. So I had no reason to change it. > The funny thing is that from time to time the replication progresses > a little. I don't like non-repetitive behavior ;) > > Thanks again for the hints. > > On 2020-04-21 14:13, Michael Menge wrote: >> Hi, >> >> Quoting Olaf Fr?czyk : >> >>> The current situation is: >>> >>> 1. Replica: >>> >>> stopped and started the replica >>> >>> no activity on replica - iotop and top show nothing >>> >>> the only messages on replica is incoming connection from master >>> >>> 2. Master: >>> >>> when I run sync_client -r I still get: >>> >>> Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file >>> /var/lib/imap/sync/log-run >>> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length >>> response to MAILBOXES (idle for too long) >>> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length >>> response to RESTART (idle for too long) >>> Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): >>> bailing out! Bad protocol >>> Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file >>> /var/lib/imap/sync/log-run failed: Bad protocol >>> >>> 3. There is 27 GB of about 45 GB replicated and there is no >>> further progress >>> >>> 4. How to find out why the replica doesn't respond? >>> >> >> You can enable telemetry logging on the replica by creating a >> folder /var/lib/imap/log/ >> where is the value of the? sync_authname. If you >> give cyrus write permissions for this folder >> it will create log-files for each process and what it received and >> send from/to the sync_client with timestamps. >> >> Also you can try to use strace on the syncserver process to figure >> out which files it is accessing >> >> >>> On 2020-04-21 11:18, Olaf Fr?czyk wrote: >>>> I also found out that when I see on master: >>>> >>>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>>> response to MAILBOXES (idle for too long) >>>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>>> response to RESTART (idle for too long) >>>> Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): >>>> bailing out! Bad protocol >>>> Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file >>>> /var/lib/imap/sync/log-run failed: Bad protocol >>>> >>>> I also get on the replica: >>>> >>>> Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, >>>> closing connection >>>> >>>> But I also see, that before it happens, there is no activity both >>>> on the replica and on the master for some time. >>>> >>>> So maybe the imap server process is not recovering correctly in >>>> the longlock situation? >>>> >>>> On 2020-04-21 11:07, Olaf Fr?czyk wrote: >>>>> Hi, >>>>> >>>>> When I run sync_client -r on the master I see the following on >>>>> the replica: >>>>> >>>>> Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock >>>>> navi.pl!user.olaf for 1.7 seconds >>>>> Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock >>>>> navi.pl!user.piotr for 2.0 seconds >>>>> Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock >>>>> navi.pl!user.olaf for 2.9 seconds >>>>> Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock >>>>> navi.pl!user.piotr for 3.0 seconds >>>>> >>>>> The mailboxes have several GB in Inbox folder and several GB in >>>>> subfolders. The inbox folders have about 20,000 messages, the >>>>> subfolders upto 15,000 >>>>> >>>>> Could it cause problems? >> >> the longlock is normaly not the problem. While on process has the lock >> no other process can write to the mailbox, but on the replica there normaly >> is no other process that should access the mailbox >> >>>>> >>>>> Maybe I should move the sync_client from START section to >>>>> SERVICES, it seems that it is not automatically restarted >>>>> >> I havend tried starting sync_client in the service section. >> I start the sync_client via systemd. >> >>>>> On 2020-04-21 08:47, Michael Menge wrote: >>>>>> Hi Olaf >>>>>> >>>>>> >>>>>> Quoting Olaf Fr?czyk : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I upgraded to 3.0.13 but it didn't help. >>>>>>> >>>>>>> This time it copied about 18GB >>>>>>> >>>>>>> in the logs I still see: >>>>>>> >>>>>>> 1 - inefficient replication >>>>>>> >>>>>>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>>>>>> IOERROR: zero length response to RESTART (idle for too long) >>>>>>> Error in do_sync(): bailing out! Bad protocol >>>>>>> >>>>>>> But I have no idea what can I do next and why it fails >>>>>>> >>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>>> response to MAILBOXES (idle for too long) >>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>>> response to RESTART (idle for too long) >>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): >>>>>>> bailing out! Bad protocol >>>>>> >>>>>> do you see any errors on the syncserver side. The error look like the >>>>>> sync_client is waiting for a reply of the server. >> >> >> -------------------------------------------------------------------------------- M.Menge??????????????????????????????? Tel.: (49) >> 7071/29-70316 >> Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 >> Zentrum f?r Datenverarbeitung????????? mail: >> michael.menge at zdv.uni-tuebingen.de >> W?chterstra?e 76 >> 72074 T?bingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From olaf at navi.pl Tue Apr 21 09:21:11 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 15:21:11 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> Message-ID: <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> I managed to get strace on both sides, however it doesn't make me wiser - there is nothing obvious for me. Additionally I see that replication works more or less for new messages, but older are not processed. I have several subfolders in my mailbox, some of them unreplicated. If I change anything in the subfolder now - the folder is replicated, but other subfolders remain not replicated - unless I change anything in them. Below strace, maybe someone can find anything meaningful from it: 1: On replica rename("/var/lib/imap/proc/6794.new", "/var/lib/imap/proc/6794") = 0 getpid()??????????????????????????????? = 6794 openat(AT_FDCWD, "/var/lib/imap/proc/6794.new", O_RDWR|O_CREAT|O_TRUNC, 0666) = 12 fstat(12, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 write(12, "imap\tifs.local.navi.pl [192.168."..., 53) = 53 close(12)?????????????????????????????? = 0 rename("/var/lib/imap/proc/6794.new", "/var/lib/imap/proc/6794") = 0 rt_sigprocmask(SIG_BLOCK, [INT QUIT ALRM TERM CHLD], [], 8) = 0 pselect6(1, [0], NULL, NULL, {tv_sec=1920, tv_nsec=0}, {[], 8}) = 1 (in [0], left {tv_sec=1919, tv_nsec=999999238}) rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT QUIT ALRM TERM CHLD], [], 8) = 0 pselect6(1, [0], NULL, NULL, {tv_sec=1920, tv_nsec=0}, {[], 8}) = 1 (in [0], left {tv_sec=1919, tv_nsec=999999439}) rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 read(0, "", 5)????????????????????????? = 0 unlink("/var/lib/imap/proc/6794")?????? = 0 openat(AT_FDCWD, "/dev/null", O_RDWR)?? = 12 shutdown(0, SHUT_RD)??????????????????? = -1 ENOTCONN (Transport endpoint is not connected) dup2(12, 0)???????????????????????????? = 0 shutdown(1, SHUT_RD)??????????????????? = -1 ENOTCONN (Transport endpoint is not connected) dup2(12, 1)???????????????????????????? = 1 shutdown(2, SHUT_RD)??????????????????? = -1 ENOTCONN (Transport endpoint is not connected) dup2(12, 2)???????????????????????????? = 2 close(12)?????????????????????????????? = 0 close(11)?????????????????????????????? = 0 getpid()??????????????????????????????? = 6794 write(3, "\1\0\0\0\212\32\0\0", 8)????? = 8 rt_sigaction(SIGALRM, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigaction(SIGQUIT, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigaction(SIGTERM, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigaction(SIGUSR2, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigaction(SIGHUP, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 alarm(83)?????????????????????????????? = 0 fcntl(8, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 stat("/usr/local/cyrus-3.0.13/libexec/imapd", {st_mode=S_IFREG|0755, st_size=1181120, ...}) = 0 rt_sigaction(SIGHUP, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT QUIT ALRM TERM CHLD], [], 8) = 0 pselect6(5, [4], NULL, NULL, NULL, {[], 8}) = ? ERESTARTNOHAND (To be restarted if no handler) --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- rt_sigreturn({mask=[INT QUIT ALRM TERM CHLD]}) = -1 EINTR (Interrupted system call) rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGHUP, {sa_handler=0x7f57e06da710, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f57dc71a960}, NULL, 8) = 0 fcntl(8, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 close(5)??????????????????????????????? = 0 munmap(0x7f57e14c1000, 16384)?????????? = 0 close(7)??????????????????????????????? = 0 munmap(0x7f57e14bd000, 16384)?????????? = 0 close(6)??????????????????????????????? = 0 unlink("/var/lib/imap/socket/idle.6794") = 0 close(9)??????????????????????????????? = 0 munmap(0x7f57e14a9000, 16384)?????????? = 0 exit_group(0)?????????????????????????? = ? +++ exited with 0 +++ 2. On master: nanosleep({1, 0}, 0x7ffe12318060)?????? = 0 stat("/var/local/imapd_sync_stop", 0x7ffe12318280) = -1 ENOENT (No such file or directory) stat("/var/lib/imap/sync/log-run", 0x7ffe12318190) = -1 ENOENT (No such file or directory) stat("/var/lib/imap/sync/log", {st_mode=S_IFREG|0600, st_size=26, ...}) = 0 rename("/var/lib/imap/sync/log", "/var/lib/imap/sync/log-run") = 0 open("/var/lib/imap/sync/log-run", O_RDWR) = 8 fcntl(8, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(8, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 read(8, "MAILBOX navi.pl!user.olaf\n", 4096) = 26 read(8, "", 4096)?????????????????????? = 0 write(6, "\27\3\3\0B\372\222\265gb\346\246$\211 \367e\241\212\30\364\345[GJ\204\225\n\255?\356\343"..., 71) = 71 rt_sigprocmask(SIG_BLOCK, [INT QUIT ALRM TERM CHLD], [], 8) = 0 pselect6(7, [6], NULL, NULL, {0, 0}, {[], 8}) = 0 (Timeout) rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 sendto(7, "<179>Apr 21 14:56:46 sync_client"..., 103, MSG_NOSIGNAL, NULL, 0) = 103 write(6, "\27\3\3\0(\372\222\265gb\346\246%\216\222#\336\16t\320\236|bv\25\261)6/\242\217l"..., 45) = 45 sendto(7, "<179>Apr 21 14:56:46 sync_client"..., 101, MSG_NOSIGNAL, NULL, 0) = 101 sendto(7, "<179>Apr 21 14:56:46 sync_client"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 sendto(7, "<179>Apr 21 14:56:46 sync_client"..., 113, MSG_NOSIGNAL, NULL, 0) = 113 close(8)??????????????????????????????? = 0 write(6, "\27\3\3\0\"\372\222\265gb\346\246&\261B\342|\260\231\217\307[\16\\\335\211\324w\345\2544\""..., 39) = 39 shutdown(6, SHUT_RD)??????????????????? = 0 close(6)??????????????????????????????? = 0 close(5)??????????????????????????????? = 0 munmap(0x7fc6ed5bb000, 16384)?????????? = 0 close(3)??????????????????????????????? = 0 munmap(0x7fc6ed5d0000, 49152)?????????? = 0 close(4)??????????????????????????????? = 0 exit_group(0)?????????????????????????? = ? +++ exited with 0 +++ On 2020-04-21 15:00, Michael Menge wrote: > > Quoting Olaf Fr?czyk : > >> Thank you for the telemetry hint :) >> >> I don't use the syncserver - the replication is done via IMAP port on >> the replica side. I have no idea how to have strace spawned by cyrus >> master process. When I attach later to imapd using strace -p I'm >> afraid some info already will be lost. >> > > You can try "/usr/bin/strace /imapd > " as cmd > or you can use prefork=1 in the service to prefok one imap process and > connect > to this one before you start the sync_client. > > > >> The syncserver is marked as deprecated in the docs, so I went with >> the more modern option. Maybe here is the problem ;) >> > > i am using syncserver since cyrus-imapd 2.3. So I had no reason to > change it. > > >> The funny thing is that from time to time the replication progresses >> a little. I don't like non-repetitive behavior ;) >> >> Thanks again for the hints. >> >> On 2020-04-21 14:13, Michael Menge wrote: >>> Hi, >>> >>> Quoting Olaf Fr?czyk : >>> >>>> The current situation is: >>>> >>>> 1. Replica: >>>> >>>> stopped and started the replica >>>> >>>> no activity on replica - iotop and top show nothing >>>> >>>> the only messages on replica is incoming connection from master >>>> >>>> 2. Master: >>>> >>>> when I run sync_client -r I still get: >>>> >>>> Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file >>>> /var/lib/imap/sync/log-run >>>> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length >>>> response to MAILBOXES (idle for too long) >>>> Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length >>>> response to RESTART (idle for too long) >>>> Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): bailing >>>> out! Bad protocol >>>> Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file >>>> /var/lib/imap/sync/log-run failed: Bad protocol >>>> >>>> 3. There is 27 GB of about 45 GB replicated and there is no further >>>> progress >>>> >>>> 4. How to find out why the replica doesn't respond? >>>> >>> >>> You can enable telemetry logging on the replica by creating a folder >>> /var/lib/imap/log/ >>> where is the value of the sync_authname. If you give >>> cyrus write permissions for this folder >>> it will create log-files for each process and what it received and >>> send from/to the sync_client with timestamps. >>> >>> Also you can try to use strace on the syncserver process to figure >>> out which files it is accessing >>> >>> >>>> On 2020-04-21 11:18, Olaf Fr?czyk wrote: >>>>> I also found out that when I see on master: >>>>> >>>>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>>>> response to MAILBOXES (idle for too long) >>>>> Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length >>>>> response to RESTART (idle for too long) >>>>> Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): >>>>> bailing out! Bad protocol >>>>> Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file >>>>> /var/lib/imap/sync/log-run failed: Bad protocol >>>>> >>>>> I also get on the replica: >>>>> >>>>> Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer, >>>>> closing connection >>>>> >>>>> But I also see, that before it happens, there is no activity both >>>>> on the replica and on the master for some time. >>>>> >>>>> So maybe the imap server process is not recovering correctly in >>>>> the longlock situation? >>>>> >>>>> On 2020-04-21 11:07, Olaf Fr?czyk wrote: >>>>>> Hi, >>>>>> >>>>>> When I run sync_client -r on the master I see the following on >>>>>> the replica: >>>>>> >>>>>> Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock >>>>>> navi.pl!user.olaf for 1.7 seconds >>>>>> Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock >>>>>> navi.pl!user.piotr for 2.0 seconds >>>>>> Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock >>>>>> navi.pl!user.olaf for 2.9 seconds >>>>>> Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock >>>>>> navi.pl!user.piotr for 3.0 seconds >>>>>> >>>>>> The mailboxes have several GB in Inbox folder and several GB in >>>>>> subfolders. The inbox folders have about 20,000 messages, the >>>>>> subfolders upto 15,000 >>>>>> >>>>>> Could it cause problems? >>> >>> the longlock is normaly not the problem. While on process has the lock >>> no other process can write to the mailbox, but on the replica there >>> normaly >>> is no other process that should access the mailbox >>> >>>>>> >>>>>> Maybe I should move the sync_client from START section to >>>>>> SERVICES, it seems that it is not automatically restarted >>>>>> >>> I havend tried starting sync_client in the service section. >>> I start the sync_client via systemd. >>> >>>>>> On 2020-04-21 08:47, Michael Menge wrote: >>>>>>> Hi Olaf >>>>>>> >>>>>>> >>>>>>> Quoting Olaf Fr?czyk : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I upgraded to 3.0.13 but it didn't help. >>>>>>>> >>>>>>>> This time it copied about 18GB >>>>>>>> >>>>>>>> in the logs I still see: >>>>>>>> >>>>>>>> 1 - inefficient replication >>>>>>>> >>>>>>>> 2 - IOERROR: zero length response to MAILBOXES (idle for too long) >>>>>>>> IOERROR: zero length response to RESTART (idle for too long) >>>>>>>> Error in do_sync(): bailing out! Bad protocol >>>>>>>> >>>>>>>> But I have no idea what can I do next and why it fails >>>>>>>> >>>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>>>> response to MAILBOXES (idle for too long) >>>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length >>>>>>>> response to RESTART (idle for too long) >>>>>>>> Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): >>>>>>>> bailing out! Bad protocol >>>>>>> >>>>>>> do you see any errors on the syncserver side. The error look >>>>>>> like the >>>>>>> sync_client is waiting for a reply of the server. >>> >>> >>> -------------------------------------------------------------------------------- >>> M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 >>> Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 >>> Zentrum f?r Datenverarbeitung????????? mail: >>> michael.menge at zdv.uni-tuebingen.de >>> W?chterstra?e 76 >>> 72074 T?bingen >>> >>> ---- >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Tue Apr 21 10:00:46 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 21 Apr 2020 16:00:46 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> Message-ID: <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> Hi, Quoting Olaf Fr?czyk : > I managed to get strace on both sides, however it doesn't make me > wiser - there is nothing obvious for me. > > Additionally I see that replication works more or less for new > messages, but older are not processed. > > I have several subfolders in my mailbox, some of them unreplicated. > If I change anything in the subfolder now - the folder is > replicated, but other subfolders remain not replicated - unless I > change anything in them. you are trying to use rolling replication. For rolling replication cyrus logs which the location and type where changes ocure e.g "MAILBOX navi.pl!user.olaf" indicates that something has changed in the INBOX of the user olaf at navi.pl. the rolling replication will keep the master and replica in sync, but it requires that there was an initial replication of all users. you can use sync_client with -A oder -u to do this synchronization (see the manpage for details) I think using rolling replication without the initial sync may be the cause of the errors. stop the "sync_client -r" and wait for the initial sync to finish. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From olaf at navi.pl Tue Apr 21 10:51:30 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 21 Apr 2020 16:51:30 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> Message-ID: <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> On 2020-04-21 16:00, Michael Menge wrote: > Hi, > > Quoting Olaf Fr?czyk : > >> I managed to get strace on both sides, however it doesn't make me >> wiser - there is nothing obvious for me. >> >> Additionally I see that replication works more or less for new >> messages, but older are not processed. >> >> I have several subfolders in my mailbox, some of them unreplicated. >> If I change anything in the subfolder now - the folder is replicated, >> but other subfolders remain not replicated - unless I change anything >> in them. > > you are trying to use rolling replication. > For rolling replication cyrus logs which the location and type where > changes ocure > e.g "MAILBOX navi.pl!user.olaf" indicates that something has changed > in the INBOX of > the user olaf at navi.pl. > > the rolling replication will keep the master and replica in sync, but > it requires > that there was an initial replication of all users. > > you can use sync_client with -A oder -u to do this synchronization > (see the manpage for details) > > I think using rolling replication without the initial sync may be the > cause of the errors. > stop the "sync_client -r" and wait for the initial sync to finish. Yes, at the beginning I was also thinking if initial sync is necessary, but there was nothing in docs about it, something started replicating and I simply assumed it does initial resync. I'll try it this evening. :) Since you use replication - are sieve scripts replicated as well? There is -s option called sieve mode but it needs to specify which users' files are to replicate and there is written that it is mostly for debugging. > > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Tue Apr 21 14:40:30 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 21 Apr 2020 20:40:30 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> Message-ID: <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> Quoting Olaf Fr?czyk : > > Yes, at the beginning I was also thinking if initial sync is > necessary, but there was nothing in docs about it, something started > replicating and I simply assumed it does initial resync. I'll try it > this evening. :) > > Since you use replication - are sieve scripts replicated as well? > There is -s option called sieve mode but it needs to specify which > users' files are to replicate and there is written that it is mostly > for debugging. > Yes, sieve scripts are replicated. The way the rolling replication works is, that every time something is changed on the master a "hint" is written in the sync log, "MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo has changed and the sync_client will sync this (and only this folder) There are other "hints" e.g for changed subscription or changed sieve script. But if the sieve script is not changed sync_client in rolling replication will not try to sync it. Using the -A or -u Option will sync the all/some users, including all mailboxes, folder subscriptions and sieve scripts. The -s option is only needed if you change a compiled sieve script so that it is not logged in the replication log. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From ellie at fastmail.com Tue Apr 21 20:36:31 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 22 Apr 2020 10:36:31 +1000 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> Message-ID: <95bc5e72-2ebe-4b60-ae54-541bfe97d28e@www.fastmail.com> I think Michael's got this pretty much covered -- you need to disable the rolling replication for now, and then use sync_client -u (or if you're brave, sync_client -A) to get an initial sync of everything. These two options work entire-user-at-a-time, so they should detect and fix the problems introduced by the partial rolling sync. If you have mailboxes in a shared namespace (i.e. that are outside the user/ namespace), they won't be replicated by -u or -A. You'll need to initially replicate those individually with sync_client -m. Once you've got a complete initial sync done, you can use rolling replication to keep the replica up to date. You can put the rolling 'sync_client -r' in the DAEMON section, so that Cyrus will restart it if it exits. Or you could manage it from outside Cyrus, e.g. via systemd/initd if you prefer. You cannot put sync_client in the SERVICES section. The SERVICES section is for service processes (i.e. processes that listen on a socket and service client requests). sync_client is a client, not a service. Cheers, ellie On Wed, Apr 22, 2020, at 4:40 AM, Michael Menge wrote: > > Quoting Olaf Fr?czyk : > > > > > Yes, at the beginning I was also thinking if initial sync is > > necessary, but there was nothing in docs about it, something started > > replicating and I simply assumed it does initial resync. I'll try it > > this evening. :) > > > > Since you use replication - are sieve scripts replicated as well? > > There is -s option called sieve mode but it needs to specify which > > users' files are to replicate and there is written that it is mostly > > for debugging. > > > > Yes, sieve scripts are replicated. > > The way the rolling replication works is, that every time something is changed > on the master a "hint" is written in the sync log, > > "MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo > has changed > and the sync_client will sync this (and only this folder) > There are other "hints" e.g for changed subscription or changed sieve script. > > But if the sieve script is not changed sync_client in rolling replication > will not try to sync it. Using the -A or -u Option will sync the all/some > users, including all mailboxes, folder subscriptions and sieve scripts. > > The -s option is only needed if you change a compiled sieve script so > that it is not logged in the replication log. > > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From lists at kwiatek.eu.org Wed Apr 22 03:16:56 2020 From: lists at kwiatek.eu.org (Andrzej Kwiatkowski) Date: Wed, 22 Apr 2020 09:16:56 +0200 Subject: Replication failed 3.0.5 -> 3.0.13 In-Reply-To: References: Message-ID: <3807490c-2c55-a2b7-b0cf-57f08fd76d3f@kwiatek.eu.org> W dniu 20.04.2020 o?16:11, Olaf Fr?czyk pisze: > Hi, > > I'm running 3.0.5. > > I want to migrate to a new machine. I set up cyrus-imapd 3.0.13. > > The replication started but it didn't transfer all mails. > > The store isn't big 44GB, transferred was about 24 GB. > > In the logs I see: > Olaf, Do you install 3.0.13 on vm or bare-metal ? Regards AK From olaf at navi.pl Wed Apr 22 04:00:41 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Wed, 22 Apr 2020 10:00:41 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> Message-ID: <724a0953-a41a-6f2b-98e5-138185814527@navi.pl> On 2020-04-21 20:40, Michael Menge wrote: > > Quoting Olaf Fr?czyk : > >> >> Yes, at the beginning I was also thinking if initial sync is >> necessary, but there was nothing in docs about it, something started >> replicating and I simply assumed it does initial resync. I'll try it >> this evening. :) >> >> Since you use replication - are sieve scripts replicated as well? >> There is -s option called sieve mode but it needs to specify which >> users' files are to replicate and there is written that it is mostly >> for debugging. >> > > Yes, sieve scripts are replicated. > > The way the rolling replication works is, that every time something is > changed > on the master a "hint" is written in the sync log, > > "MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo > has changed > and the sync_client will sync this (and only this folder) > There are other "hints" e.g for changed subscription or changed sieve > script. > > But if the? sieve script is not changed sync_client in rolling > replication > will not try to sync it. Using the -A or -u Option will sync the all/some > users, including all mailboxes, folder subscriptions and sieve scripts. > > The -s option is only needed if you change a compiled sieve script so > that it is not logged in the replication log. I did replication with -A. It looks that everything works properly now. Sieve scripts were also transferred :) It is good to know, that if I change compiled script manually it needs additional attention. Thank you very much for help :) > > > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From olaf at navi.pl Wed Apr 22 04:04:22 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Wed, 22 Apr 2020 10:04:22 +0200 Subject: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13 In-Reply-To: <95bc5e72-2ebe-4b60-ae54-541bfe97d28e@www.fastmail.com> References: <20200421084718.Horde.3DkfgnS6gWHjh562A3fpvvA@webmail.uni-tuebingen.de> <0dcf0171-45c0-1b09-e8fa-3bfc74a02c0b@navi.pl> <20200421141305.Horde.n7T4_Dk7qd_BrPW1R5cFuOJ@webmail.uni-tuebingen.de> <20200421150056.Horde.GPNxBu15nGgagwv8Xek5llp@webmail.uni-tuebingen.de> <1861f746-10fa-3ae9-b1be-187f58daf800@navi.pl> <20200421160046.Horde.bVv4jbzEm2bCiCtn_XQZSAr@webmail.uni-tuebingen.de> <5e055c98-e65b-dfbc-c0bd-c862a03b6db7@navi.pl> <20200421204030.Horde.j57-8_eazzgzIhc0U5miBbW@webmail.uni-tuebingen.de> <95bc5e72-2ebe-4b60-ae54-541bfe97d28e@www.fastmail.com> Message-ID: <952c6054-d290-deb9-ad2b-d40523bf4454@navi.pl> Yes, Michael was right - it works properly now. (I hope ;). OK. I'll put it in the DAEMON section - this way I have only one point where all stuff related to imap is started. Thank you for explanation. Regards, Olaf On 2020-04-22 02:36, ellie timoney wrote: > I think Michael's got this pretty much covered -- you need to disable the rolling replication for now, and then use sync_client -u (or if you're brave, sync_client -A) to get an initial sync of everything. These two options work entire-user-at-a-time, so they should detect and fix the problems introduced by the partial rolling sync. > > If you have mailboxes in a shared namespace (i.e. that are outside the user/ namespace), they won't be replicated by -u or -A. You'll need to initially replicate those individually with sync_client -m. > > Once you've got a complete initial sync done, you can use rolling replication to keep the replica up to date. You can put the rolling 'sync_client -r' in the DAEMON section, so that Cyrus will restart it if it exits. Or you could manage it from outside Cyrus, e.g. via systemd/initd if you prefer. > > You cannot put sync_client in the SERVICES section. The SERVICES section is for service processes (i.e. processes that listen on a socket and service client requests). sync_client is a client, not a service. > > Cheers, > > ellie > > On Wed, Apr 22, 2020, at 4:40 AM, Michael Menge wrote: >> Quoting Olaf Fr?czyk : >> >>> Yes, at the beginning I was also thinking if initial sync is >>> necessary, but there was nothing in docs about it, something started >>> replicating and I simply assumed it does initial resync. I'll try it >>> this evening. :) >>> >>> Since you use replication - are sieve scripts replicated as well? >>> There is -s option called sieve mode but it needs to specify which >>> users' files are to replicate and there is written that it is mostly >>> for debugging. >>> >> Yes, sieve scripts are replicated. >> >> The way the rolling replication works is, that every time something is changed >> on the master a "hint" is written in the sync log, >> >> "MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo >> has changed >> and the sync_client will sync this (and only this folder) >> There are other "hints" e.g for changed subscription or changed sieve script. >> >> But if the sieve script is not changed sync_client in rolling replication >> will not try to sync it. Using the -A or -u Option will sync the all/some >> users, including all mailboxes, folder subscriptions and sieve scripts. >> >> The -s option is only needed if you change a compiled sieve script so >> that it is not logged in the replication log. >> >> >> >> >> -------------------------------------------------------------------------------- >> M.Menge Tel.: (49) 7071/29-70316 >> Universit?t T?bingen Fax.: (49) 7071/29-5912 >> Zentrum f?r Datenverarbeitung mail: >> michael.menge at zdv.uni-tuebingen.de >> W?chterstra?e 76 >> 72074 T?bingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From olaf at navi.pl Wed Apr 22 04:19:40 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Wed, 22 Apr 2020 10:19:40 +0200 Subject: Replication failed 3.0.5 -> 3.0.13 In-Reply-To: <3807490c-2c55-a2b7-b0cf-57f08fd76d3f@kwiatek.eu.org> References: <3807490c-2c55-a2b7-b0cf-57f08fd76d3f@kwiatek.eu.org> Message-ID: On 2020-04-22 09:16, Andrzej Kwiatkowski wrote: > W dniu 20.04.2020 o?16:11, Olaf Fr?czyk pisze: >> Hi, >> >> I'm running 3.0.5. >> >> I want to migrate to a new machine. I set up cyrus-imapd 3.0.13. >> >> The replication started but it didn't transfer all mails. >> >> The store isn't big 44GB, transferred was about 24 GB. >> >> In the logs I see: >> > Olaf, > > Do you install 3.0.13 on vm or bare-metal ? > > Regards > AK I have it installed in VM - CentOS 7, now 8. This setup worked OK for many years. But I have only a few users, just with big mailboxes. On SAS drives it works fine. The only longlock I saw was during the replication - I suppose transferring 20.000 big messages at once per mailbox could cause it. But in normal usage I don't see it. The cause of my problem was my ignorance regarding how rolling replication works in cyrus. It works OK now :) Regards, Olaf From falon at ruparpiemonte.it Tue Apr 28 06:49:52 2020 From: falon at ruparpiemonte.it (Marco) Date: Tue, 28 Apr 2020 12:49:52 +0200 Subject: METADATA help Message-ID: <7cf8b667-aca7-d832-804f-44bae93ff3c1@ruparpiemonte.it> Hello, I have some problems in Cyrus 3.0.13 with user defined metadata. Let suppose I define in imapd.conf annotation_definitions: /etc/annoIMAP.conf and annoIMAP.conf is: /vendor/example/partition/comune.prova.it,server,string,backend,value.priv,lrswipkxtea With this configuration all works fine with the old annotatemore "standard". I can set and read annotations. But time will crawl, so I have to adopt the new METADATA standard (RFC5464) to manage these values. If I try a SETMETADATA "" (/private/vendor/example/partition/comune.prova.it "prova1") a NO Permission denied I have to do this to set the value: a SETANNOTATION "" "/vendor/example/partition/comune.prova.it" ("value.priv" "prova1") a OK Completed Now, with a GETMETADATA "" "/private/*" * METADATA "" ("/private/vendor/example/partition/comune.prova.it" "prova1" "/private/vendor/cmu/cyrus-imapd/squat" NIL "/private/vendor/cmu/cyrus-imapd/expire" NIL "/private/vendor/cmu/cyrus-imapd/usercounters" "3 0 0 0 0 0 0 0 0 0 0" "/private/vendor/cmu/cyrus-imapd/usermodseq" "0" "/private/comment" NIL "/private/admin" NIL) but if I ask only my user defined metadata: a GETMETADATA "" "/private/vendor/example/partition/comune.prova.it" * METADATA "" ("/private/vendor/example/partition/comune.prova.it" NIL) the server tell me NIL in place of "prova1". I would ask an help with this protocol: - I would like to know why SETMETADATA fails with "Permission denied" in above example. - Then I have to know why a wildcard query show me the "prova1" value in "/private/vendor/example/partition/comune.prova.it", but when I query only the value of "/private/vendor/example/partition/comune.prova.it" the answer is NIL. Other detail: name : Cyrus IMAPD version : 3.0.13-6.el8 vendor : Project Cyrus support-url: https://www.cyrusimap.org os : Linux os-version : 4.18.0-147.8.1.el8_1.x86_64 environment: Built w/Cyrus SASL 2.1.27 Running w/Cyrus SASL 2.1.27 Built w/OpenSSL 1.1.1c FIPS 28 May 2019 Running w/OpenSSL 1.1.1c FIPS 28 May 2019 Built w/zlib 1.2.11 Running w/zlib 1.2.11 CMU Sieve 3.0 NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = idled Thank you very much Kind Regards Marco From olaf at navi.pl Tue Apr 28 11:12:13 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Tue, 28 Apr 2020 17:12:13 +0200 Subject: sync_log_chain - is it always needed? Message-ID: <9727a706-6f3f-1086-9700-b0d9ce072fd6@navi.pl> Hello, I was wondering why do we need to use this option on middle servers in replication chain? I don't use sync_server. The replication is done using IMAP protocol. Is in this case this setting also necessary? Shouldn't the middle server catch all changes that are done via IMAP protocol and forward them to the next server in the replication chain? BTW. Does anyone has some description how sieve scripts replication is done via IMAP protocol? Best regards, OIaf From ellie at fastmail.com Tue Apr 28 21:39:24 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 29 Apr 2020 11:39:24 +1000 Subject: sync_log_chain - is it always needed? In-Reply-To: <9727a706-6f3f-1086-9700-b0d9ce072fd6@navi.pl> References: <9727a706-6f3f-1086-9700-b0d9ce072fd6@navi.pl> Message-ID: <5cac835c-e5fa-48e9-9f15-54de800df678@www.fastmail.com> On Wed, Apr 29, 2020, at 1:12 AM, Olaf Fr?czyk wrote: > I was wondering why do we need to use this option on middle servers in > replication chain? When a user/delivery/admin/etc action happens on a mailbox/user/etc, an entry is recorded in the sync log saying that something has happened to that mailbox/user/etc, and that it will need to be checked and replicated at some point. sync_client in rolling mode (or log-replay mode) reads that sync log, and for each mailbox/user/etc that the sync log reports as having been changed, it compares its current state against the replica, and updates the replica to match if necessary. When a change arrives by replication, it is NOT logged to the sync_log -- unless sync_log_chain has been enabled. So if you want a rolling replication arrangement such that Master -> Replica1 -> Replica2, then Replica1 _must_ have sync_log_chain enabled, otherwise there will never be anything in its sync log, and it will therefore never forward the replications on to Replica2. This arrangement has the advantage that Master only needs to do one replication (and can focus its energy on servicing clients instead), but the disadvantage that the state on Replica2 will not be as fresh as the state on Master or Replica1. But if you want an arrangement such that Master -> Replica1 and Master -> Replica2, then you don't need sync_log_chain anywhere. This arrangement has the disadvantage that Master must do more replication work, but the advantage that Replica1 and Replica2 are more likely to both be up-to-date at any given moment. > I don't use sync_server. The replication is done using IMAP protocol. Is > in this case this setting also necessary? It makes no difference; replication is replication. The replication support in imapd is a thin wrapper around the exact same implementation as used by sync_server. > Shouldn't the middle server catch all changes that are done via IMAP > protocol and forward them to the next server in the replication chain? The changes are not done "via IMAP protocol", they are done "via replication protocol, over an IMAP connection". For example, you couldn't use Cyrus replication to update some other non-Cyrus IMAP server -- it would not understand the replication commands. If you want a chained configuration, then you need sync_log_chain, regardless of whether you use imapd or sync_server. > BTW. Does anyone has some description how sieve scripts replication is > done via IMAP protocol? Can you clarify what you want to know here? It's replicated the same way as it would be to sync_server: The sync_client queries the replica's state, decides whether it needs updating, and if so, then it sends the new state, and checks for an OK response from the replica. Depending on the Cyrus version of your master server, it might (in older versions) try to send both the sieve script file and the compiled bytecode file; or (in recent versions) just the script file. Depending on the Cyrus version of your replica servers, it might accept and update a bytecode file as-is; or (in recent versions) simply accept and ignore it, and recompile the bytecode on demand from the script file that was also sent. If, for some reason, you're replicating from a recent version (that only sends the script file) to an old version (that expects both), then your replica will not have up to date compiled bytecode. If your replica needs to be brought into service taking real traffic, you will first need to manually (re)compile all your scripts using sievec. This is almost certainly not the only quirk of replicating to an older server and then trying to run it with real traffic, so try to not need to do that! :) It's not safe/possible to run bytecode on one Cyrus version that was compiled by a different Cyrus version, which is why in recent versions we do not bother to replicate bytecode at all. Recent Cyrus versions instead have a mechanism to recompile bytecode on demand if it's missing, out of date, or the wrong version, so that you will always be running the correct, up-to-date bytecode for the current Cyrus version for the current contents of the script. Regardless of your Cyrus version, the state of "which script is currently activated for the user" is also replicated in the same way. Hope this helps, ellie From olaf at navi.pl Wed Apr 29 10:29:04 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Wed, 29 Apr 2020 16:29:04 +0200 Subject: sync_log_chain - is it always needed? In-Reply-To: <5cac835c-e5fa-48e9-9f15-54de800df678@www.fastmail.com> References: <9727a706-6f3f-1086-9700-b0d9ce072fd6@navi.pl> <5cac835c-e5fa-48e9-9f15-54de800df678@www.fastmail.com> Message-ID: On 2020-04-29 03:39, ellie timoney wrote: > On Wed, Apr 29, 2020, at 1:12 AM, Olaf Fr?czyk wrote: >> I was wondering why do we need to use this option on middle servers in >> replication chain? > Hope this helps, Yes it helped. Thank you. As the replication uses its own protocol even over IMAP connection it makes sense. However why the sync_log_chain could be not always active? As the server catches all changes from LMTP, IMAP, POP anyway, why to use a special option for the synchronization protocol? Why to treat replication changes differently from LMTP/IMAP? If we want replication - we want to catch all changes - regardless of the source, if we don't do replication - we don't need to catch any changes. > > ellie > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From lists at kwiatek.eu.org Wed Apr 29 10:47:56 2020 From: lists at kwiatek.eu.org (Andrzej Kwiatkowski) Date: Wed, 29 Apr 2020 16:47:56 +0200 Subject: Replication failed 3.0.5 -> 3.0.13 In-Reply-To: References: <3807490c-2c55-a2b7-b0cf-57f08fd76d3f@kwiatek.eu.org> Message-ID: W dniu 22.04.2020 o?10:19, Olaf Fr?czyk pisze: > On 2020-04-22 09:16, Andrzej Kwiatkowski wrote: >> W dniu 20.04.2020 o?16:11, Olaf Fr?czyk pisze: >>> Hi, >>> >>> I'm running 3.0.5. >>> >>> I want to migrate to a new machine. I set up cyrus-imapd 3.0.13. >>> >>> The replication started but it didn't transfer all mails. >>> >>> The store isn't big 44GB, transferred was about 24 GB. >>> >>> In the logs I see: >>> >> Olaf, >> >> Do you install 3.0.13 on vm or bare-metal ? >> >> Regards >> AK > > I have it installed in VM - CentOS 7, now 8. This setup worked OK for > many years. But I have only a few users, just with big mailboxes. > > On SAS drives it works fine. The only longlock I saw was during the > replication - I suppose transferring 20.000 big messages at once per > mailbox could cause it. But in normal usage I don't see it. > > The cause of my problem was my ignorance regarding how rolling > replication works in cyrus. It works OK now :) > > Regards, > > Olaf > Ok. I was asking because of problem with low entropy on VM-s causing performance issues with big installations. I've also noticed that replication documentation is a bit tricky, and not every procedure/case is well documented. Regards AK From olaf at navi.pl Wed Apr 29 11:38:38 2020 From: olaf at navi.pl (=?UTF-8?Q?Olaf_Fr=c4=85czyk?=) Date: Wed, 29 Apr 2020 17:38:38 +0200 Subject: Replication failed 3.0.5 -> 3.0.13 In-Reply-To: References: <3807490c-2c55-a2b7-b0cf-57f08fd76d3f@kwiatek.eu.org> Message-ID: On 2020-04-29 16:47, Andrzej Kwiatkowski wrote: > Ok. > > I was asking because of problem with low entropy on VM-s causing > performance issues with big installations. I didn't have this issue, however as I said, my installation is really small. If I needed more entropy I would think of using a hardware entropy generator. Eg. on a PCI-e card. They aren't super cheap but also not too expensive. I think someone with good cryptography knowledge could help you with this topic. I suppose the storage I/O can be a bigger issue here. > > I've also noticed that replication documentation is a bit tricky, > and not every procedure/case is well documented. Yes, me too ;) But if you hit a bigger problem, you have this list :) I tried on a copy without active users to be safe. > > Regards > AK From egoitz at sarenet.es Wed Apr 29 11:58:56 2020 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Wed, 29 Apr 2020 17:58:56 +0200 Subject: Mailbox rename and Cyrus murder Message-ID: Hi, Sometimes, we need to redistribute mailboxes between different Cyrus partitions in the same server. We do it by blocking previously any kind of accesses to the mailbox and after stopping any kind of active connection to that mailbox. I was wondering, if this exactly, is really needed because it?s really something that could cause problems to the mailbox (we do it this way always, as a procedure for avoiding problems, but is it really needed??). I?m in the process of automating this kind of operation, but reading some info before writing code, I have seen that Cyrus murder allows ?live? mailbox migration? perhaps is this another option for avoid doing mailbox rename operations for moving a mailbox to a different partition?. I have read too (perhaps have not understand properly), that a Cyrus Murder can redirect requests to another mailbox server, in case one of them fails?. Is it like a automated way of changing master/slave roles?. How is your experience with murders?. Do they allow this what I have asked?. If they do have you ever seen any kind of issue with it?, Cheers, From falon at ruparpiemonte.it Thu Apr 30 06:41:34 2020 From: falon at ruparpiemonte.it (Marco) Date: Thu, 30 Apr 2020 12:41:34 +0200 Subject: METADATA help In-Reply-To: <7cf8b667-aca7-d832-804f-44bae93ff3c1@ruparpiemonte.it> References: <7cf8b667-aca7-d832-804f-44bae93ff3c1@ruparpiemonte.it> Message-ID: <9fc39fde-d036-7fd6-95a5-ea12a1d9a0c2@ruparpiemonte.it> Oh, I have found the solution! It seems that in the annotation_definitions file the *name* of the third-party annotation MUST be down-case. The RFC5464 says: "Except as noted otherwise, all alphabetic characters are case- insensitive. The use of upper or lower case characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion." So this could be a bug... Interesting this: if I define a downcase name in the file, then the GET/SETMETADATA works case insensitive as expected. But if the name in annotation file is uppercase, the imap metadata commands return NIL or Permission denied. Kind Regards Marco Il 28/04/2020 12:49, Marco ha scritto: > Hello, > > ? I have some problems in Cyrus 3.0.13 with user defined metadata. > > Let suppose I define in imapd.conf > > ????annotation_definitions: /etc/annoIMAP.conf > > and annoIMAP.conf is: > > /vendor/example/partition/comune.prova.it,server,string,backend,value.priv,lrswipkxtea > > > With this configuration all works fine with the old annotatemore > "standard". I can set and read annotations. > > But time will crawl, so I have to adopt the new METADATA standard > (RFC5464) to manage these values. > > If I try > > a SETMETADATA "" (/private/vendor/example/partition/comune.prova.it > "prova1") > a NO Permission denied > > I have to do this to set the value: > > a SETANNOTATION "" "/vendor/example/partition/comune.prova.it" > ("value.priv" "prova1") > a OK Completed > > > Now, with > > a GETMETADATA "" "/private/*" > * METADATA "" ("/private/vendor/example/partition/comune.prova.it" > "prova1" "/private/vendor/cmu/cyrus-imapd/squat" NIL > "/private/vendor/cmu/cyrus-imapd/expire" NIL > "/private/vendor/cmu/cyrus-imapd/usercounters" "3 0 0 0 0 0 0 0 0 0 0" > "/private/vendor/cmu/cyrus-imapd/usermodseq" "0" "/private/comment" NIL > "/private/admin" NIL) > > but if I ask only my user defined metadata: > a GETMETADATA "" "/private/vendor/example/partition/comune.prova.it" > * METADATA "" ("/private/vendor/example/partition/comune.prova.it" NIL) > > the server tell me NIL in place of "prova1". > > I would ask an help with this protocol: > > - I would like to know why SETMETADATA fails with "Permission denied" in > above example. > > - Then I have to know why a wildcard query show me the "prova1" value in > "/private/vendor/example/partition/comune.prova.it", but when I query > only the value of "/private/vendor/example/partition/comune.prova.it" > the answer is NIL. > > Other detail: > name?????? : Cyrus IMAPD > version??? : 3.0.13-6.el8 > vendor???? : Project Cyrus > support-url: https://www.cyrusimap.org > os???????? : Linux > os-version : 4.18.0-147.8.1.el8_1.x86_64 > environment: Built w/Cyrus SASL 2.1.27 > ???????????? Running w/Cyrus SASL 2.1.27 > ???????????? Built w/OpenSSL 1.1.1c FIPS? 28 May 2019 > ???????????? Running w/OpenSSL 1.1.1c FIPS? 28 May 2019 > ???????????? Built w/zlib 1.2.11 > ???????????? Running w/zlib 1.2.11 > ???????????? CMU Sieve 3.0 > ???????????? NET-SNMP > ???????????? mmap = shared > ???????????? lock = fcntl > ???????????? nonblock = fcntl > ???????????? idle = idled > > > Thank you very much > Kind Regards > > Marco > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > From egoitz at sarenet.es Thu Apr 30 06:43:12 2020 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Thu, 30 Apr 2020 12:43:12 +0200 Subject: Mailbox rename and Cyrus murder In-Reply-To: References: Message-ID: <41EEB9D2-B8FB-430D-A1B6-A3DB9F6C8CD4@sarenet.es> Hi!, I have been reading this interesting thread https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2015-March/003165.html from 2015. As a curiosity, did finally that ended in some sort of production env?. I have seen this kind of pages https://www.cyrusimap.org/imap/concepts/features/server-aggregation.html but have not seen their publish date or similar? I?m in the state at which, operations in mail servers are being more and more frequent? they are starting not being able to be done without automation, but before that.. and writing code? I wanted to know if someone could share some experience in horizontal scaling (backend or mailbox server clustering), because live moving mailboxes without interruption or in the most possible easy way (just as examples), are important things to know before writing all the automation stuff? knowing if a Cyrus Murder could failover from backend automatically (after a crash) or similar? Reading at present https://www.cyrusimap.org/imap/concepts/features/server-aggregation.html ? Any shared experience would be highly appreciated, Cheers! Egoitz Aurrekoetxea Dpto. de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > El 29 abr 2020, a las 17:58, egoitz at sarenet.es escribi?: > > Hi, > > Sometimes, we need to redistribute mailboxes between different Cyrus partitions in the same server. We do it by blocking previously any kind of accesses to the mailbox and after stopping any kind of active connection to that mailbox. I was wondering, if this exactly, is really needed because it?s really something that could cause problems to the mailbox (we do it this way always, as a procedure for avoiding problems, but is it really needed??). I?m in the process of automating this kind of operation, but reading some info before writing code, I have seen that Cyrus murder allows ?live? mailbox migration? perhaps is this another option for avoid doing mailbox rename operations for moving a mailbox to a different partition?. > > I have read too (perhaps have not understand properly), that a Cyrus Murder can redirect requests to another mailbox server, in case one of them fails?. Is it like a automated way of changing master/slave roles?. > > How is your experience with murders?. Do they allow this what I have asked?. If they do have you ever seen any kind of issue with it?, > > Cheers, > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From falon at ruparpiemonte.it Thu Apr 30 08:31:27 2020 From: falon at ruparpiemonte.it (Marco) Date: Thu, 30 Apr 2020 14:31:27 +0200 Subject: conversation db home Message-ID: <7ccd426d-30ce-a4e1-9e5d-b4543582bfc6@ruparpiemonte.it> Hello, I have some doubts, maybe you could help me to understand. I see these user-level files in the "configdirectory" path: domain//.[conversations|counters|sub|xapianactive] Why don't these files stay in the metapartition- path? My config is: metapartition_files: header index cache expunge squat annotations lock dav archivecache No option for conversation. And the xapianactive file is in the configdirectory even if the "squat" is named in the list of "metapartition_files". In particular the conversation db seems to become very large, so we could better manage the fs space, if it could stay in the metapartition path. In other words, I wouldn't like to see the domain/ path in configdirectory. Is there a way to achieve this? Other detail: name : Cyrus IMAPD version : 3.0.13-6.el8 vendor : Project Cyrus support-url: https://www.cyrusimap.org os : Linux os-version : 4.18.0-147.8.1.el8_1.x86_64 environment: Built w/Cyrus SASL 2.1.27 Running w/Cyrus SASL 2.1.27 Built w/OpenSSL 1.1.1c FIPS 28 May 2019 Running w/OpenSSL 1.1.1c FIPS 28 May 2019 Built w/zlib 1.2.11 Running w/zlib 1.2.11 CMU Sieve 3.0 NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = idled #defaultpartition: default #partition-default: /var/spool/imap partition-maildata4: /maildata/example.com/maildata1 partition-maildata40: /maildata/example.com/maildata2 archivepartition-maildata4: /sysarchivio/example.com/maildata1 archivepartition-maildata40: /sysarchivio/example.com/maildata2 metapartition-maildata4: /metamaildata/example.com/maildata1 metapartition-maildata40: /metamaildata/example.com/maildata2 # Metapartition files metapartition_files: header index cache expunge squat annotations lock dav archivecache Thank you very much Kind Regards Marco From ellie at fastmail.com Thu Apr 30 19:26:12 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 01 May 2020 09:26:12 +1000 Subject: sync_log_chain - is it always needed? In-Reply-To: References: <9727a706-6f3f-1086-9700-b0d9ce072fd6@navi.pl> <5cac835c-e5fa-48e9-9f15-54de800df678@www.fastmail.com> Message-ID: On Thu, Apr 30, 2020, at 12:29 AM, Olaf Fr?czyk wrote: > However why the sync_log_chain could be not always active? As the server > catches all changes from LMTP, IMAP, POP anyway, why to use a special > option for the synchronization protocol? Why to treat replication > changes differently from LMTP/IMAP? If we want replication - we want to > catch all changes - regardless of the source, if we don't do replication > - we don't need to catch any changes. The rolling sync_client takes care of cleaning up each sync_log file as it finishes replicating it downstream. Now consider the case where your replica is an end point, not a link in a chain: it does not have a rolling sync_client forwarding replications on, so if it automatically logged incoming replications to the sync_log, the sync_log would simply grow forever and fill the disk. You would need to set up a special job to delete it.... Better to simply not write it in the first place! Thus, the default is to sync_log_chain: off, and if you need the special-case chaining behaviour, you turn it on. Cheers, ellie