From a_s_y at sama.ru Wed Jul 1 06:52:00 2020 From: a_s_y at sama.ru (Sergey) Date: Wed, 1 Jul 2020 14:52:00 +0400 Subject: Cyrus IMAP 3.2.2 released In-Reply-To: <81e56b4e-4ba6-43ae-a5b0-0af1d304fc0e@www.fastmail.com> References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com> <202006301610.52081.a_s_y@sama.ru> <81e56b4e-4ba6-43ae-a5b0-0af1d304fc0e@www.fastmail.com> Message-ID: <202007011452.00037.a_s_y@sama.ru> On Wednesday 01 July 2020, ellie timoney wrote: > Yes, please. If you don't, I will, but then you won't get > automatic notifications of updates. https://github.com/cyrusimap/cyrus-imapd/issues/3090 > Are they appearing in a log somewhere, or is this output > from an analysis tool you're running? Second. This is ALT Linux specific part of rpm-build package: http://git.altlinux.org/gears/r/rpm-build.git This is a heavily modified part of the original rpm 4.0.4, but verify-elf is a sh script and it can be used separately. 3 sh scripts from rpm-build is needed: verify-elf, ldd and functions. -- Regards, Sergey From falon at ruparpiemonte.it Wed Jul 1 09:57:45 2020 From: falon at ruparpiemonte.it (Marco) Date: Wed, 1 Jul 2020 15:57:45 +0200 Subject: backupd and sync_client IOERROR In-Reply-To: References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> Message-ID: <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> On 19/06/2020 03:01, ellie timoney has written: > Rolling mode only makes incremental updates, so if you're starting from a server that already has existing data, you should do the first manual initial backups before enabling the rolling mode. Hello, about this error, I retried more times. It seems really reproducible all times. I don't enable rolling mode. At the initial sync if I do: $ sync_client -A -n bck -z -v USER dan at example.com MAILBOX example.com!user.dan MAILBOX example.com!user.dan.Sent QUOTA example.com!user.dan QUOTA example.com!user.dan.Sent USER din at example.com MAILBOX example.com!user.din MAILBOX example.com!user.din.Spam MAILBOX example.com!user.din.Trash QUOTA example.com!user.din USER don at example.com MAILBOX example.com!user.don QUOTA example.com!user.don USER utente.archivista at example.com MAILBOX example.com!user.utente^archivista Error from do_user(utente.archivista at example.com): bailing out! and the error log is: 2020-07-01T15:42:14.161553+02:00 tst-msg03 cyrus/sync_client[4861]: MESSAGE received NO response: IMAP_PROTOCOL_ERROR Protocol error 2020-07-01T15:42:14.296022+02:00 tst-msg03 cyrus/sync_client[4861]: do_folders(): update failed: example.com!user.utente^archivista 'Bad protocol' 2020-07-01T15:42:14.296220+02:00 tst-msg03 cyrus/sync_client[4861]: IOERROR: do_user_main: Bad protocol for utente.archivista at example.com to [no channel] (tst-msg03-bck.example.com) 2020-07-01T15:42:14.296290+02:00 tst-msg03 cyrus/sync_client[4861]: Error in do_user(utente.archivista at example.com): bailing out! If I repeat the "sync_client -A -n bck -z -v" the error still occurs every time. If I do: $ sync_client -n bck -z -v -u utente.archivista at example.com USER utente.archivista at example.com MAILBOX example.com!user.utente^archivista MAILBOX example.com!user.utente^archivista.Cest MAILBOX example.com!user.utente^archivista.Posta archiviata MAILBOX example.com!user.utente^archivista.Posta archiviata.Archivio [...] it works without errors. Uhm... Thank you again Cheers Marco From jesper at schmitz.computer Thu Jul 2 15:23:36 2020 From: jesper at schmitz.computer (Jesper Schmitz Mouridsen) Date: Thu, 2 Jul 2020 21:23:36 +0200 Subject: UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder. Message-ID: Hi. I recently upgraded Cyrus to 3.2.2 from 3.0.13. Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with >50K mails, makes imapd process use 100 CPU% without any progress. truss[1] or dtrace shows no output. The process seems totally stuck. I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of how to use gdb to see what? is going one. I could not make imapd run under gdb even with the -D option and debug_command setting or reading https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug It is fast enough on other folders also with ~50k mails < 3 secs. Any hints, highly appreciated :-) [1] https://www.freebsd.org/cgi/man.cgi?truss Regards Jesper From ellie at fastmail.com Thu Jul 2 19:52:33 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 03 Jul 2020 09:52:33 +1000 Subject: UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder. In-Reply-To: References: Message-ID: Hi, I think I would do something like: 0) set client_timeout to a big value (see below) 1) let the imapd start normally 2) connect to it with a minimal imap client (like imtest or telnet) 3) check logs to see which imapd process id your client is connected to (if there's more than one) 4) use the "gdb /path/to/binary pid" invocation to attach gdb to the running imap process 5) repeat #4 in a few different ways if it's unable to find symbols 6) set a breakpoint on the cmd_thread function (since that's what handles the THREAD command) and then continue 7) back in your imap client, send the UID THREAD REFS US-ASCII ALL" command and step through to see what happens Note that these are not exhaustive steps, more of a "get started, and then react accordingly, depending on what you see" I would also try variations of that THREAD command -- if you add/remove options, does it start working? Can you isolate the problem to a specific combination of options? Does it only happen for some mailboxes? You probably want to set client_timeout to a big value. The default is 10 seconds, so the server will probably throw your client off while you're reading output from gdb, and then you'll wind up debugging the "disconnect an unresponsive client code" instead. I usually set this to be 30 minutes or so for debugging. For 3.2 and newer, you can spell this as "30m". For older versions, the value is in seconds, so you want "1800". Permissions may be awkward. You will need to be the cyrus user (or root) to connect gdb to the running imapd, but you will also need permission to read the source that it was built from, which is probably not owned by the cyrus user. In my case it's under my user account's home directory, so I add the cyrus user to my user account's group, and make sure the path to the source directory is g+rx (because I don't like running gdb as root). Cheers, ellie On Fri, Jul 3, 2020, at 5:23 AM, Jesper Schmitz Mouridsen via Info-cyrus wrote: > Hi. > > I recently upgraded Cyrus to 3.2.2 from 3.0.13. > > Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with > >50K mails, > > makes imapd process use 100 CPU% without any progress. truss[1] or dtrace > > shows no output. The process seems totally stuck. > > I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of > how to use gdb to > > see what? is going one. I could not make imapd run under gdb even with > the -D option > > and debug_command setting or reading > https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug > > It is fast enough on other folders also with ~50k mails < 3 secs. > > Any hints, highly appreciated :-) > > [1] https://www.freebsd.org/cgi/man.cgi?truss > > Regards > > Jesper > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From ellie at fastmail.com Thu Jul 2 23:08:11 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 03 Jul 2020 13:08:11 +1000 Subject: backupd and sync_client IOERROR In-Reply-To: <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> Message-ID: <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> On Wed, Jul 1, 2020, at 11:57 PM, Marco wrote: > Uhm... Wow, that's wierd. I notice that the users that worked correctly with "sync_client -A" don't have dots in their address localparts. If you create another user that also has a dot, does it fail under -A in the same way? Does it fail in the same way replicating to a normal replica, instead of a backup server? From ismael.tanguy at univ-brest.fr Fri Jul 3 04:25:17 2020 From: ismael.tanguy at univ-brest.fr (=?UTF-8?Q?Isma=c3=abl_Tanguy?=) Date: Fri, 3 Jul 2020 10:25:17 +0200 Subject: sync_client: Fallback on master Message-ID: Hello, During the upgrade of a master backend server (~15k mailboxes) from Centos7 to Centos8, we fail over on a replica. Before the failover, we run a last time sync_client -r, to flush entirely sync log. Everything went fine, except that the replica was not allowed to mupdatepush (ctl_mboxlist -m) and all the mailboxes dissapear in the murder mailboxes.db That has been fixed as soon as we saw this error. Mailboxes.db has been repopulated. Then, when the backend cames up, we fall back on it and run sync_client from the replica to the master. I think now that was not the good way, because new mails start to arrive on the master while replication occured from the replica. Replica's sync log size is as big as yesterday (the failover day) = 23M and around 1 million lines (mainly MAILBOX and APPEND commands). replica /var/log/maillog reports various errors : Jul? 3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: UNMAILBOX received NO response: Unknown error Jul? 3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: sync_folder_delete(): failed: user.e21901967.Bibli-Culture 'The remote Server(s) denied the operation' Jul? 3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: IOERROR: do_user_main: The remote Server(s) denied the operation for e21901967 to [no channel] (store-xxxxx) Jul? 3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: Error in do_sync(): bailing out! The remote Server(s) denied the operation Jul? 3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: SYNCNOTICE: highestmodseq higher on replica user.e21603781, updating 8692 => 8694 Jul? 3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: SYNCNOTICE: record mismatch with replica: user.e21603781 more recent on replica Jul? 3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: SYNCNOTICE: master uid:4340 modseq:8692 last_updated:1593736158 internaldate:1593736091 flags:(\Seen) cid:0000000000000000 Jul? 3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: SYNCNOTICE: replica uid:4340 modseq:8693 last_updated:1593762055 internaldate:1593736091 flags:(\Seen) cid:0000000000000000 Jul? 3 09:43:03 store-1-replicat-etudiant cyrus/sync_client[4045]: MAILBOX received NO response: IMAP_SYNC_CHECKSUM Checksum Failure Jul? 3 09:43:03 store-1-replicat-etudiant cyrus/sync_client[4045]: CRC failure on sync for user.e21913149, trying full update Therefore, there are 24Go of additionals mails on the master IMAP spool, compared to the replica. What's is the best way to empty the replica sync_log? I would like to be sure that every mail arrived on the replica during the failover has been copied on the master. I could use external tool like imapsync, maybe Cyrus has one that permits that. Thank you, -- Isma?l Tanguy From jesper at schmitz.computer Sat Jul 4 07:35:18 2020 From: jesper at schmitz.computer (Jesper Schmitz Mouridsen) Date: Sat, 4 Jul 2020 13:35:18 +0200 Subject: UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder. In-Reply-To: References: Message-ID: <455f7b37-6860-4c3b-5aa9-c40ac3b13e13@schmitz.computer> Hi Thanks for the debugging hints! client_timeout sat to 30M and the UID THREAD REFS US-ASCII ALL actually completes. But first after ~10 mins on a CPU: Intel(R) Celeron(R) CPU N2930 @ 1.83GHz (1833.38-MHz K8-class CPU) and after 69.484 secs on a CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (The same jail, moved to faster hardware) 63994 messages in around 40k threads. What could cause the much spent time building the thread data? version is still 3.2.2, but 3.0.13 shows the same behavior. The thread output of the slow folder is here FWIW https://pastebin.com/PpiJ8X2E Regards /Jesper On 03.07.2020 01.52, ellie timoney wrote: > Hi, > > I think I would do something like: > > 0) set client_timeout to a big value (see below) > 1) let the imapd start normally > 2) connect to it with a minimal imap client (like imtest or telnet) > 3) check logs to see which imapd process id your client is connected to (if there's more than one) > 4) use the "gdb /path/to/binary pid" invocation to attach gdb to the running imap process > 5) repeat #4 in a few different ways if it's unable to find symbols > 6) set a breakpoint on the cmd_thread function (since that's what handles the THREAD command) and then continue > 7) back in your imap client, send the UID THREAD REFS US-ASCII ALL" command and step through to see what happens > > Note that these are not exhaustive steps, more of a "get started, and then react accordingly, depending on what you see" > > I would also try variations of that THREAD command -- if you add/remove options, does it start working? Can you isolate the problem to a specific combination of options? Does it only happen for some mailboxes? > > You probably want to set client_timeout to a big value. The default is 10 seconds, so the server will probably throw your client off while you're reading output from gdb, and then you'll wind up debugging the "disconnect an unresponsive client code" instead. I usually set this to be 30 minutes or so for debugging. For 3.2 and newer, you can spell this as "30m". For older versions, the value is in seconds, so you want "1800". > > Permissions may be awkward. You will need to be the cyrus user (or root) to connect gdb to the running imapd, but you will also need permission to read the source that it was built from, which is probably not owned by the cyrus user. In my case it's under my user account's home directory, so I add the cyrus user to my user account's group, and make sure the path to the source directory is g+rx (because I don't like running gdb as root). > > Cheers, > > ellie > > On Fri, Jul 3, 2020, at 5:23 AM, Jesper Schmitz Mouridsen via Info-cyrus wrote: >> Hi. >> >> I recently upgraded Cyrus to 3.2.2 from 3.0.13. >> >> Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with >> >50K mails, >> >> makes imapd process use 100 CPU% without any progress. truss[1] or dtrace >> >> shows no output. The process seems totally stuck. >> >> I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of >> how to use gdb to >> >> see what? is going one. I could not make imapd run under gdb even with >> the -D option >> >> and debug_command setting or reading >> https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug >> >> It is fast enough on other folders also with ~50k mails < 3 secs. >> >> Any hints, highly appreciated :-) >> >> [1] https://www.freebsd.org/cgi/man.cgi?truss >> >> Regards >> >> Jesper >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From falon at ruparpiemonte.it Mon Jul 6 10:17:53 2020 From: falon at ruparpiemonte.it (Marco) Date: Mon, 6 Jul 2020 16:17:53 +0200 Subject: backupd and sync_client IOERROR In-Reply-To: <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> Message-ID: Hello, On 03/07/2020 05:08, ellie timoney has written: > I notice that the users that worked correctly with "sync_client -A" don't have dots in their address localparts. If you create another user that also has a dot, does it fail under -A in the same way? I copied the content of the failing mailbox into another mailbox with a name without dots: springsteen at example.com and the result is the same: # sync_client -A -n bck -z -v USER springsteen at example.com MAILBOX example.com!user.springsteen Error from do_user(springsteen at example.com): bailing out! 2020-07-06T15:31:51.538935+02:00 tst-msg03-bck backupd[2234574]: login: tst-msg03.example.com [10.102.102.102] cyr_backup LOGIN User logged in 2020-07-06T15:31:51.576283+02:00 tst-msg03-bck backupd[2234574]: creating sql_db /var/spool/cyr_backup/s/springsteen at example.com_t8Yjcb.index 2020-07-06T15:36:57.552290+02:00 tst-msg03 cyrus/sync_client[17488]: MESSAGE received NO response: IMAP_PROTOCOL_ERROR Protocol error 2020-07-06T15:36:57.753294+02:00 tst-msg03 cyrus/sync_client[17488]: do_folders(): update failed: example.com!user.springsteen 'Bad protocol' 2020-07-06T15:36:57.758121+02:00 tst-msg03 cyrus/sync_client[17488]: IOERROR: do_user_main: Bad protocol for springsteen at example.com to [no channel] (tst-msg03-bck.example.com) 2020-07-06T15:36:57.765868+02:00 tst-msg03 cyrus/sync_client[17488]: Error in do_user(springsteen at example.com): bailing out! Instead, if I do: # sync_client -n bck -z -v -u springsteen at uc.csi.it USER springsteen at example.com MAILBOX example.com!user.springsteen MAILBOX example.com!user.springsteen.#splitconversations MAILBOX example.com!user.springsteen.ALLARMI MAILBOX example.com!user.springsteen.ALLARMI.John [...] the program replicates as well and without any errors. > Does it fail in the same way replicating to a normal replica, instead of a backup server? At this moment I don't have a normal replica server yet. I'll try... Thank you very much Cheers Marco From ellie at fastmail.com Wed Jul 8 00:23:12 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 08 Jul 2020 14:23:12 +1000 Subject: backupd and sync_client IOERROR In-Reply-To: References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> Message-ID: Hi Marco, On Tue, Jul 7, 2020, at 12:17 AM, Marco wrote: > I copied the content of the failing mailbox into another mailbox with a > name without dots: Oh that's very curious. It suggests that something about the mailbox contents is causing it to fail under -A but not under -u. I was hoping it would just be the dot in the name, cause something like that should be fairly easy to reproduce and debug. But it's sounding like it'll be more difficult! It might be worth running "mbexamine" over some of these mailboxes -- both some that work correctly, and the ones that are misbehaving -- and see if any interesting patterns emerge? The man page is here: https://www.cyrusimap.org/imap/reference/manpages/systemcommands/mbexamine.html > > Does it fail in the same way replicating to a normal replica, instead of a backup server? > > At this moment I don't have a normal replica server yet. I'll try... Thanks. I think this will be important for figuring out whether the problem is replication generally, or backupd specific. Cheers, ellie From falon at ruparpiemonte.it Wed Jul 8 08:08:15 2020 From: falon at ruparpiemonte.it (Marco) Date: Wed, 8 Jul 2020 14:08:15 +0200 Subject: backupd and sync_client IOERROR In-Reply-To: References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> Message-ID: Hi Ellie, On 08/07/2020 06:23, ellie timoney has written: > Oh that's very curious. It suggests that something about the mailbox contents is causing it to fail under -A but not under -u. I was hoping it would just be the dot in the name, cause something like that should be fairly easy to reproduce and debug. But it's sounding like it'll be more difficult! > > It might be worth running "mbexamine" over some of these mailboxes -- both some that work correctly, and the ones that are misbehaving -- and see if any interesting patterns emerge? The man page is here:https://www.cyrusimap.org/imap/reference/manpages/systemcommands/mbexamine.html The mailbox which fails is quite large (over 32000 msgs and over 1GB). mbexamine produces a very large output for all mails. I can see that it doesn't fail: it terminates without errors and with 0 exit status. Even switches -c or -q terminates without errors. All quotas checks are correct. >>> Does it fail in the same way replicating to a normal replica, instead of a backup server? >> At this moment I don't have a normal replica server yet. I'll try... > Thanks. I think this will be important for figuring out whether the problem is replication generally, or backupd specific. I understand. As soon as I can, I'll setup a normal replica. Thank you Cheers Marco From steffo76 at gmx.de Thu Jul 9 05:47:37 2020 From: steffo76 at gmx.de (Stephan) Date: Thu, 9 Jul 2020 11:47:37 +0200 Subject: lmtp Trying to unput wrong character Message-ID: Hello, I am having trouble with a mail stuck in the queue, it can't get delivered because lmtpd has some problem, it says: lmtp: FATAL: Trying to unput wrong character I'm not sure what to do about this. I found the error message in prot_ungetc() which is sitting in lib/prot.c but don't know what this actually does. If I interpret this correctly, it gets called in pushmsg() in lmtpengine.c and the comment says that there is a carriage return that should be put back, which apparently doesn't work. Any hints ? Thanks, Stephan From Hagedorn at uni-koeln.de Thu Jul 9 07:45:22 2020 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 9 Jul 2020 13:45:22 +0200 Subject: lmtp Trying to unput wrong character In-Reply-To: References: Message-ID: <21b8a15e-d0e9-52be-5e76-36c5cbb7a330@uni-koeln.de> Am 09.07.20 um 11:47 Uhr schrieb Stephan: > Hello, > > I am having trouble with a mail stuck in the queue, it can't get > delivered because lmtpd has some problem, it says: > > lmtp: FATAL: Trying to unput wrong character > > I'm not sure what to do about this. I found the error message in > prot_ungetc() which is sitting in lib/prot.c but don't know what this > actually does. If I interpret this correctly, it gets called in > pushmsg() in lmtpengine.c and the comment says that there is a carriage > return that should be put back, which apparently doesn't work. > > Any hints ? If you check the list archives you will find that this happens occasionally. IMO your best bet is to delete the mail from the queue. Cheers, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: Hagedorn.vcf Type: text/x-vcard Size: 333 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5367 bytes Desc: S/MIME Cryptographic Signature URL: From ismael.tanguy at univ-brest.fr Fri Jul 10 09:00:56 2020 From: ismael.tanguy at univ-brest.fr (=?UTF-8?Q?Isma=c3=abl_Tanguy?=) Date: Fri, 10 Jul 2020 15:00:56 +0200 Subject: IOERROR: writing cache file : Bad address Message-ID: Hello, we have separate our Postfix Server from our old Cyrus Server and so pass LMTP on TCP : ? #lmtpunix???? cmd="/usr/cyrus/bin/lmtpd -C /etc/imapd.conf" listen="/var/lib/imap/socket/lmtp" prefork=10 maxchild=240 ?lmtp????????? cmd="lmtpd" listen="lmtp" prefork=1 We are not facing problems with seen State reseting all Inbox mails to Unread. We have many lines like that in /var/log/messages : ?skiplist: checkpointed /var/lib/imap/user/s/silaot.seen (43 records, 3160 bytes) in 0 seconds and this ones : IOERROR: writing cache file for user.silaot.Trash: Bad address We have observed that all mails came back to unread after expunging a mail. We've tried with no success to reconstruct mailbox folder after delete cyrus files (cyrus.cache, cyrus.index, ..) Has anyone some advices to help us? Our cyrus version is : > version name?????? : Cyrus IMAPD version??? : v2.3.11 2007/12/10 14:48:14 vendor???? : Project Cyrus support-url: http://cyrusimap.web.cmu.edu os???????? : Linux os-version : 2.6.18-194.11.4.el5 environment: Built w/Cyrus SASL 2.1.22 ???????????? Running w/Cyrus SASL 2.1.22 ???????????? Built w/Sleepycat Software: Berkeley DB 4.3.29: (January? 7, 2007) ???????????? Running w/Sleepycat Software: Berkeley DB 4.3.29: (January? 7, 2007) ???????????? Built w/OpenSSL 0.9.8b 04 May 2006 ???????????? Running w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 ???????????? CMU Sieve 2.3 ???????????? TCP Wrappers ???????????? mmap = shared ???????????? lock = fcntl ???????????? nonblock = fcntl ???????????? idle = idled Thanks, Isma?l TANGUY -- From Hagedorn at uni-koeln.de Tue Jul 14 10:52:38 2020 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Tue, 14 Jul 2020 16:52:38 +0200 Subject: www.cyrusimap.org is down Message-ID: <3873e8a8-163d-e157-99f2-5b8118aeb1a4@uni-koeln.de> Hi, currently the hostname www.cyrusimap.org cannot be resolved. cyrusimap.org still works, but it appears to point to Github, from where there is a redirect to www.cyrusimap.org -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5367 bytes Desc: S/MIME Cryptographic Signature URL: From ellie at fastmail.com Tue Jul 14 20:39:33 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 15 Jul 2020 10:39:33 +1000 Subject: www.cyrusimap.org is down In-Reply-To: <3873e8a8-163d-e157-99f2-5b8118aeb1a4@uni-koeln.de> References: <3873e8a8-163d-e157-99f2-5b8118aeb1a4@uni-koeln.de> Message-ID: It's hosted on GitHub Pages. There's supposed to be a DNS CNAME entry at "www.cyrusimap.org" pointing to "cyrusimap.github.io", but it seems to have disappeared. /sigh Without that DNS entry, even going directly to "cyrusimap.github.io" isn't working, because GitHub Pages wants to redirect to our custom domain... which is currently broken. Thanks for the alert, we'll try and get hold of someone at CMU to get this fixed (or maybe they're watching the list and see this thread). Cheers, ellie On Wed, Jul 15, 2020, at 12:52 AM, Sebastian Hagedorn wrote: > Hi, > > currently the hostname www.cyrusimap.org cannot be resolved. > cyrusimap.org still works, but it appears to point to Github, from where > there is a redirect to www.cyrusimap.org > -- > .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. > .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * smime.p7s -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Tue Jul 14 23:27:24 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 15 Jul 2020 13:27:24 +1000 Subject: www.cyrusimap.org is down In-Reply-To: References: <3873e8a8-163d-e157-99f2-5b8118aeb1a4@uni-koeln.de> Message-ID: CMU report that it's been fixed, and it does indeed seem to be working again for me now. The usual caveats about DNS propagation probably apply. I don't know why the entry disappeared, and if the cause was something automated I suppose it might disappear again... I guess we'll keep an eye on it and see what unfolds. Cheers, ellie On Wed, Jul 15, 2020, at 10:39 AM, ellie timoney wrote: > It's hosted on GitHub Pages. There's supposed to be a DNS CNAME entry at "www.cyrusimap.org" pointing to "cyrusimap.github.io", but it seems to have disappeared. /sigh > > Without that DNS entry, even going directly to "cyrusimap.github.io" isn't working, because GitHub Pages wants to redirect to our custom domain... which is currently broken. > > Thanks for the alert, we'll try and get hold of someone at CMU to get this fixed (or maybe they're watching the list and see this thread). > > Cheers, > > ellie > > On Wed, Jul 15, 2020, at 12:52 AM, Sebastian Hagedorn wrote: >> Hi, >> >> currently the hostname www.cyrusimap.org cannot be resolved. >> cyrusimap.org still works, but it appears to point to Github, from where >> there is a redirect to www.cyrusimap.org >> -- >> .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. >> .:.Regionales Rechenzentrum (RRZK).:. >> .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. >> >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> *Attachments:* >> * smime.p7s > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.s at aptalaska.net Wed Jul 15 21:23:18 2020 From: matt.s at aptalaska.net (Matthew Schumacher) Date: Wed, 15 Jul 2020 18:23:18 -0700 Subject: assertion failed: imap/mboxevent.c: 743: filled_params(type, event) Message-ID: <8a833bd3-617f-b7fe-730c-6a107aa8183c@aptalaska.net> I'm trying to use external notifications on 3.2.2 but it doesn't work.?? If I define event_notifier: external notify_external: /usr/cyrus/bin/cyrus_notify event_groups: access event_extra_params: clientAddress timestamp service Then the imapd thread dies with this assertion: Jul 15 18:01:54 snow imaps[28108]: Cannot notify event Login: missing parameters: serverAddress clientAddress Jul 15 18:01:54 snow imaps[28108]: Fatal error: Internal error: assertion failed: imap/mboxevent.c: 743: filled_params(type, event) If I remove event_groups and event_extra_params then notify never calls my external script and notify breaks. If I define "event_groups: access" and omit event_extra_params then I'm back to: Jul 15 18:14:26 snow imaps[28934]: Cannot notify event Login: missing parameters: serverAddress Jul 15 18:14:26 snow imaps[28934]: Fatal error: Internal error: assertion failed: imap/mboxevent.c: 743: filled_params(type, event) Anyone know where this serverAddress is coming from and how to fix it? schu From ellie at fastmail.com Thu Jul 16 20:47:40 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 17 Jul 2020 10:47:40 +1000 Subject: =?UTF-8?Q?Re:_assertion_failed:_imap/mboxevent.c:_743:_filled=5Fparams(t?= =?UTF-8?Q?ype,_event)?= In-Reply-To: <8a833bd3-617f-b7fe-730c-6a107aa8183c@aptalaska.net> References: <8a833bd3-617f-b7fe-730c-6a107aa8183c@aptalaska.net> Message-ID: <7bdc2f85-ca54-454d-8092-a431f454b004@www.fastmail.com> Hi, I've seen something like this before, and my gut feel is that this is going to turn out to be a bug in Cyrus. I think what's happening is that, somewhere in Cyrus, an event is being generated with a type that's supposed to contain a serverAddress field, but the serverAddress field is not being initialised. Before a generated event actually gets sent out to the notifier, we validate that all the required parameters have been filled ("filled_params()"), and the fatal assertion is telling us that this one has not been, even though it should have been. Would you like to open a GitHub issue at https://github.com/cyrusimap/cyrus-imapd/issues ? If you don't, I will. But if I do it, then you won't get automatic notifications about updates, so if you can, it's better if you do it. Feel free to just paste your previous email as the issue text. :) Cheers, ellie On Thu, Jul 16, 2020, at 11:23 AM, Matthew Schumacher wrote: > I'm trying to use external notifications on 3.2.2 but it doesn't work.?? > If I define > > event_notifier: external > notify_external: /usr/cyrus/bin/cyrus_notify > event_groups: access > event_extra_params: clientAddress timestamp service > > Then the imapd thread dies with this assertion: > > Jul 15 18:01:54 snow imaps[28108]: Cannot notify event Login: missing > parameters: serverAddress clientAddress > Jul 15 18:01:54 snow imaps[28108]: Fatal error: Internal error: > assertion failed: imap/mboxevent.c: 743: filled_params(type, event) > > If I remove event_groups and event_extra_params then notify never calls > my external script and notify breaks. > > If I define "event_groups: access" and omit event_extra_params then I'm > back to: > > Jul 15 18:14:26 snow imaps[28934]: Cannot notify event Login: missing > parameters: serverAddress > Jul 15 18:14:26 snow imaps[28934]: Fatal error: Internal error: > assertion failed: imap/mboxevent.c: 743: filled_params(type, event) > > Anyone know where this serverAddress is coming from and how to fix it? > > schu > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From matt.s at aptalaska.net Fri Jul 17 04:52:31 2020 From: matt.s at aptalaska.net (Matthew Schumacher) Date: Fri, 17 Jul 2020 01:52:31 -0700 Subject: assertion failed: imap/mboxevent.c: 743: filled_params(type, event) In-Reply-To: <7bdc2f85-ca54-454d-8092-a431f454b004@www.fastmail.com> References: <8a833bd3-617f-b7fe-730c-6a107aa8183c@aptalaska.net> <7bdc2f85-ca54-454d-8092-a431f454b004@www.fastmail.com> Message-ID: <32985df9-8c2b-fd82-cb60-977241fb8895@aptalaska.net> HI Ellie, I agree that it's probably a bug.? I'll open a github issue. I'll report back with the issue number. Thanks, Matt On 7/16/20 5:47 PM, ellie timoney wrote: > Hi, > > I've seen something like this before, and my gut feel is that this is going to turn out to be a bug in Cyrus. > > I think what's happening is that, somewhere in Cyrus, an event is being generated with a type that's supposed to contain a serverAddress field, but the serverAddress field is not being initialised. > > Before a generated event actually gets sent out to the notifier, we validate that all the required parameters have been filled ("filled_params()"), and the fatal assertion is telling us that this one has not been, even though it should have been. > > Would you like to open a GitHub issue at https://github.com/cyrusimap/cyrus-imapd/issues ? If you don't, I will. But if I do it, then you won't get automatic notifications about updates, so if you can, it's better if you do it. Feel free to just paste your previous email as the issue text. :) > > Cheers, > > ellie > > On Thu, Jul 16, 2020, at 11:23 AM, Matthew Schumacher wrote: >> I'm trying to use external notifications on 3.2.2 but it doesn't work. >> If I define >> >> event_notifier: external >> notify_external: /usr/cyrus/bin/cyrus_notify >> event_groups: access >> event_extra_params: clientAddress timestamp service >> >> Then the imapd thread dies with this assertion: >> >> Jul 15 18:01:54 snow imaps[28108]: Cannot notify event Login: missing >> parameters: serverAddress clientAddress >> Jul 15 18:01:54 snow imaps[28108]: Fatal error: Internal error: >> assertion failed: imap/mboxevent.c: 743: filled_params(type, event) >> >> If I remove event_groups and event_extra_params then notify never calls >> my external script and notify breaks. >> >> If I define "event_groups: access" and omit event_extra_params then I'm >> back to: >> >> Jul 15 18:14:26 snow imaps[28934]: Cannot notify event Login: missing >> parameters: serverAddress >> Jul 15 18:14:26 snow imaps[28934]: Fatal error: Internal error: >> assertion failed: imap/mboxevent.c: 743: filled_params(type, event) >> >> Anyone know where this serverAddress is coming from and how to fix it? >> >> schu >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From matt.s at aptalaska.net Mon Jul 20 17:15:03 2020 From: matt.s at aptalaska.net (Matthew Schumacher) Date: Mon, 20 Jul 2020 14:15:03 -0700 Subject: assertion failed: imap/mboxevent.c: 743: filled_params(type, event) In-Reply-To: <32985df9-8c2b-fd82-cb60-977241fb8895@aptalaska.net> References: <8a833bd3-617f-b7fe-730c-6a107aa8183c@aptalaska.net> <7bdc2f85-ca54-454d-8092-a431f454b004@www.fastmail.com> <32985df9-8c2b-fd82-cb60-977241fb8895@aptalaska.net> Message-ID: <75133483-49f8-01dd-e29f-e4a60c2ec1eb@aptalaska.net> Bug filed: https://github.com/cyrusimap/cyrus-imapd/issues/3115 On 7/17/20 1:52 AM, Matthew Schumacher wrote: > HI Ellie, > > I agree that it's probably a bug.? I'll open a github issue. > > I'll report back with the issue number. > > Thanks, > Matt > > On 7/16/20 5:47 PM, ellie timoney wrote: >> Hi, >> >> I've seen something like this before, and my gut feel is that this is >> going to turn out to be a bug in Cyrus. >> >> I think what's happening is that, somewhere in Cyrus, an event is >> being generated with a type that's supposed to contain a >> serverAddress field, but the serverAddress field is not being >> initialised. >> >> Before a generated event actually gets sent out to the notifier, we >> validate that all the required parameters have been filled >> ("filled_params()"), and the fatal assertion is telling us that this >> one has not been, even though it should have been. >> >> Would you like to open a GitHub issue at >> https://github.com/cyrusimap/cyrus-imapd/issues ?? If you don't, I >> will.? But if I do it, then you won't get automatic notifications >> about updates, so if you can, it's better if you do it.? Feel free to >> just paste your previous email as the issue text. :) >> >> Cheers, >> >> ellie >> >> On Thu, Jul 16, 2020, at 11:23 AM, Matthew Schumacher wrote: >>> I'm trying to use external notifications on 3.2.2 but it doesn't work. >>> If I define >>> >>> event_notifier: external >>> notify_external: /usr/cyrus/bin/cyrus_notify >>> event_groups: access >>> event_extra_params: clientAddress timestamp service >>> >>> Then the imapd thread dies with this assertion: >>> >>> Jul 15 18:01:54 snow imaps[28108]: Cannot notify event Login: missing >>> parameters: serverAddress clientAddress >>> Jul 15 18:01:54 snow imaps[28108]: Fatal error: Internal error: >>> assertion failed: imap/mboxevent.c: 743: filled_params(type, event) >>> >>> If I remove event_groups and event_extra_params then notify never calls >>> my external script and notify breaks. >>> >>> If I define "event_groups: access" and omit event_extra_params then I'm >>> back to: >>> >>> Jul 15 18:14:26 snow imaps[28934]: Cannot notify event Login: missing >>> parameters: serverAddress >>> Jul 15 18:14:26 snow imaps[28934]: Fatal error: Internal error: >>> assertion failed: imap/mboxevent.c: 743: filled_params(type, event) >>> >>> Anyone know where this serverAddress is coming from and how to fix it? >>> >>> schu >>> ---- >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From falon at ruparpiemonte.it Wed Jul 22 05:51:01 2020 From: falon at ruparpiemonte.it (Marco) Date: Wed, 22 Jul 2020 11:51:01 +0200 Subject: backupd and sync_client Syntax error In-Reply-To: References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> Message-ID: Hello, I would ask an help with a new replication error in backup channel. I have this never-ending loop error: 2020-07-22T11:35:16.212126+02:00 tst-msg03 cyrus/sync_client[22465]: MAILBOXES received NO response: IMAP_PROTOCOL_BAD_PARAMETERS Bad parameters 2020-07-22T11:35:16.212185+02:00 tst-msg03 cyrus/sync_client[22465]: Error in do_sync(): bailing out! Syntax error in parameters 2020-07-22T11:35:16.212228+02:00 tst-msg03 cyrus/sync_client[22465]: Processing sync log file /var/lib/imap/sync/bck/log-run failed: Syntax error in parameters 2020-07-22T11:35:16.216962+02:00 tst-msg03 cyrus/sync_client[22465]: Reprocessing sync log file /var/lib/imap/sync/bck/log-run The "log-run" file ends with: APPEND "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa" MAILBOX "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa" MAILBOX "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa" APPEND "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa.Sottocartella" MAILBOX "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa.Sottocartella" MAILBOX "example.com!user.gianni^ferromagnetic. Cartella Molto Spaziosa.Sottocartella" APPEND example.com!user.gianni^ferromagnetic.10 MAILBOX example.com!user.gianni^ferromagnetic.10 MAILBOX example.com!user.gianni^ferromagnetic.10 APPEND example.com!user.gianni^ferromagnetic.Abbcc MAILBOX example.com!user.gianni^ferromagnetic.Abbcc MAILBOX example.com!user.gianni^ferromagnetic.Abbcc APPEND example.com!user.gianni^ferromagnetic.Abc MAILBOX example.com!user.gianni^ferromagnetic.Abc MAILBOX example.com!user.gianni^ferromagnetic.Abc MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1" APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Spam" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Spam" APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia.Archivio" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia.Archivio" APPEND "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" MAILBOX "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" The log-run has the same last-modified date of the first syslog error message of syntax error. Other strange folder names I created are MAILBOX example.com!user.baraccone.prova:prova: MAILBOX example.com!user.baraccone.prova:prova: MAILBOX example.com!user.baraccone.prova^1 MAILBOX example.com!user.baraccone.prova^1 I suspect that some folder name breaks the backup replication... Thank you very much Cheers Marco From falon at ruparpiemonte.it Wed Jul 22 09:13:57 2020 From: falon at ruparpiemonte.it (Marco) Date: Wed, 22 Jul 2020 15:13:57 +0200 Subject: sync_client Sieve from 2.4.20 to 3.2.2 issue Message-ID: <56a9bcda-6e42-cc63-43ee-696de3e80472@ruparpiemonte.it> Hello, I have a replication issue from Cyrus IMAP 2.4.20 to 3.2.2 about sieve path. Let suppose there is a user in Cyrus IMAP 2.4.20 with # mbpath user/gianni.ferromagnetic at example.com /maildata/example.com/maildata1/domain/C/example.com/S/user/gianni^ferromagnetic and the Sieve is in: /var/lib/imap/sieve/domain/C/example.com/S/gianni^ferromagnetic when the sync_client replicates the user in the Cyrus IMAP 3.2.2 I see: $ mbpath -a user/gianni.ferromagnetic at example.com Archive: /sysarchivio/example.com/maildata1/domain/C/example.com/H/user/gianni^ferromagnetic Data: /maildata/example.com/maildata1/domain/C/example.com/H/user/gianni^ferromagnetic Meta: /metamaildata/example.com/maildata1/domain/C/example.com/H/user/gianni^ferromagnetic Sieve: /var/spool/sieve/domain/C/example.com/H/gianni.ferromagnetic So the hash path changes. Archive, data and meta path exists really as expected. But /var/spool/sieve/domain/C/example.com/H/gianni.ferromagnetic doesn't exist. The sync_client create instead /var/spool/sieve/domain/C/example.com/S/gianni.ferromagnetic The "S" is the same hash result in master server. So it seems that for all data but sieve the new path is honored. For Sieve the new path "C" is reported by mbpath, but sync_client replicates the sieve scripts elsewhere. Both master and replica have fulldirhash: 1 hashimapspool: true unixhierarchysep: yes and the partition names are equals. If I try to open a sieve connection I see: $ telnet 0 4190 Trying 0.0.0.0... Connected to 0. Escape character is '^]'. NO Fatal error: Error initializing actions Connection closed by foreign host. and the syslog says: cyrus/sieve[31393]: can't use home directories I think this is a consequence of the above issue, uhm... Thank you very much Cheers Marco From falon at ruparpiemonte.it Wed Jul 22 10:12:43 2020 From: falon at ruparpiemonte.it (Marco) Date: Wed, 22 Jul 2020 16:12:43 +0200 Subject: sync_client Sieve from 2.4.20 to 3.2.2 issue In-Reply-To: <56a9bcda-6e42-cc63-43ee-696de3e80472@ruparpiemonte.it> References: <56a9bcda-6e42-cc63-43ee-696de3e80472@ruparpiemonte.it> Message-ID: <8731692f-2560-b51e-63b1-729ca3ea8fce@ruparpiemonte.it> > For Sieve the new path "C" > is reported by mbpath, but sync_client replicates the sieve scripts > elsewhere. Sorry, I would mean "For Sieve the new path with "H" is reported by mbpath, but sync_client replicates the scripts elsewhere (in the path with "S"). Many many thanks for every help Cheers Marco From falon at ruparpiemonte.it Thu Jul 23 05:25:23 2020 From: falon at ruparpiemonte.it (Marco) Date: Thu, 23 Jul 2020 11:25:23 +0200 Subject: backupd and sync_client Syntax error In-Reply-To: References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it> <009d8add-7726-f766-7209-4b8144d9afe1@ruparpiemonte.it> <64ac8a0f-7076-4107-96d9-ffbf415c4f91@www.fastmail.com> Message-ID: <4eabb41e-ee6d-e01d-21e5-134ab49e8e93@ruparpiemonte.it> Ouch, sorry, ignore this. I suppose to have found the problem. My goal is to sync from Cyrus IMAP 2.4 to 3.2, and to 3.2 to a backupd host. I see it works just starting by hand the sync_client in Cyrus IMAP 2.4, meanwhile in the middle host the rolling mode is sufficient with sync_log_chain. But I had a bad Cyrus IMAP 2.4 with some mailboxes which had references in mailboxes.db only, and not in file system and metadata. Solving above error, then sync_client from 3.2 IMAP host to backupd host works. I don't know why I don't see errors in sync_client from 2.4 to 3.2 already. Cheers Marco Il 22/07/2020 11:51, Marco ha scritto: > Hello, > > ?I would ask an help with a new replication error in backup channel. > I have this never-ending loop error: > > 2020-07-22T11:35:16.212126+02:00 tst-msg03 cyrus/sync_client[22465]: > MAILBOXES received NO response: IMAP_PROTOCOL_BAD_PARAMETERS Bad parameters > 2020-07-22T11:35:16.212185+02:00 tst-msg03 cyrus/sync_client[22465]: > Error in do_sync(): bailing out! Syntax error in parameters > 2020-07-22T11:35:16.212228+02:00 tst-msg03 cyrus/sync_client[22465]: > Processing sync log file /var/lib/imap/sync/bck/log-run failed: Syntax > error in parameters > 2020-07-22T11:35:16.216962+02:00 tst-msg03 cyrus/sync_client[22465]: > Reprocessing sync log file /var/lib/imap/sync/bck/log-run > > > The "log-run" file ends with: > > APPEND "example.com!user.gianni^ferromagnetic.? Cartella Molto Spaziosa" > MAILBOX "example.com!user.gianni^ferromagnetic.? Cartella Molto Spaziosa" > MAILBOX "example.com!user.gianni^ferromagnetic.? Cartella Molto Spaziosa" > APPEND "example.com!user.gianni^ferromagnetic.? Cartella Molto > Spaziosa.Sottocartella" > MAILBOX "example.com!user.gianni^ferromagnetic.? Cartella Molto > Spaziosa.Sottocartella" > MAILBOX "example.com!user.gianni^ferromagnetic.? Cartella Molto > Spaziosa.Sottocartella" > APPEND example.com!user.gianni^ferromagnetic.10 > MAILBOX example.com!user.gianni^ferromagnetic.10 > MAILBOX example.com!user.gianni^ferromagnetic.10 > APPEND example.com!user.gianni^ferromagnetic.Abbcc > MAILBOX example.com!user.gianni^ferromagnetic.Abbcc > MAILBOX example.com!user.gianni^ferromagnetic.Abbcc > APPEND example.com!user.gianni^ferromagnetic.Abc > MAILBOX example.com!user.gianni^ferromagnetic.Abc > MAILBOX example.com!user.gianni^ferromagnetic.Abc > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1" > APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino" > APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Cestino.Test" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Spam" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.Spam" > APPEND "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia.Archivio" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella 1.ma,cia.Archivio" > APPEND "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" > MAILBOX "example.com!user.gianni^ferromagnetic.Cartella Spaziosa" > > The log-run has the same last-modified date of the first syslog error > message of syntax error. > > > Other strange folder names I created are > > MAILBOX example.com!user.baraccone.prova:prova: > MAILBOX example.com!user.baraccone.prova:prova: > MAILBOX example.com!user.baraccone.prova^1 > MAILBOX example.com!user.baraccone.prova^1 > > > I suspect that some folder name breaks the backup replication... > > Thank you very much > > Cheers > Marco > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > From falon at ruparpiemonte.it Thu Jul 23 10:00:40 2020 From: falon at ruparpiemonte.it (Marco) Date: Thu, 23 Jul 2020 16:00:40 +0200 Subject: Cyrus replica 2.4 to 3.2.2 issues Message-ID: <2e45800c-b966-184f-1e93-6db26035a50f@ruparpiemonte.it> Hello, I'm trying to replicate from Cyrus IMAP 2.4.20 to Cyrus IMAP 3.2.2 using replication. Because Cyrus 2.4 doesn't support IMAP replication (is this true?), I configured the replica with replica2to3syncserver cmd="/usr/libexec/cyrus-imapd/sync_server" proto=tcp4 listen="csync" in SERVICES of cyrus.conf. The sync_client in master works in rolling mode, such as /usr/lib/cyrus-imapd/sync_client -r -n replica2-3 Both master and replica hosts work with delayed deleted and delayed expunge fashion. I experience some problems. In the below logs tst-msg01 is the master, tst-msg03 is the replica. == 1 == I found "errors" of this kind: 2020-07-21T15:05:16.803765+02:00 tst-msg03 cyrus/replica2to3syncserver[1238 sync_log(): Failed to lock /var/lib/imap/sync/squatter/log for MAILBOX example.com!user.demo^gruppoarchivio.ticket.Vecchi#012 after 65 attempts 2020-07-21T15:05:16.803898+02:00 tst-msg03 cyrus/replica2to3syncserver[1238 mailbox: longlock example.com!user.demo^gruppoarchivio.ticket.Vecchi for 2.5 seconds 2020-07-21T15:05:17.090124+02:00 tst-msg03 cyrus/replica2to3syncserver[1238 sync_log(): Failed to lock /var/lib/imap/sync/squatter/log for MAILBOX example.com!user.demo^gruppoarchivio.ticket.Vecchi#012 after 65 attempts 2020-07-21T15:05:17.101288+02:00 tst-msg03 cyrus/replica2to3syncserver[1238 mappedfile: longlock /var/lib/imap/domain/Q/example.com/user/E/demo.gruppoarchivio.conversations for 2.7 seconds 2020-07-21T15:05:17.101410+02:00 tst-msg03 cyrus/squatter[27276]: indexing mailbox user/demo.gruppoarchivio/ticket/Vecchi at example.com... 2020-07-21T15:05:18.132371+02:00 tst-msg03 cyrus/squatter[27276]: Xapian committed 740 updates in 0.374236 sec 2020-07-21T15:05:18.145507+02:00 tst-msg03 cyrus/squatter[27276]: mappedfile: longlock /var/lib/imap/domain/Q/example.com/user/E/demo.gruppoarchivio.xapianactive for 1.0 seconds But the xapian index seems to be ok: -rw------- 1 cyrus mail 864 Jul 21 15:05 cyrus.indexed.db -rw------- 1 cyrus mail 0 Jul 21 15:05 flintlock -rw------- 1 cyrus mail 116 Jul 21 15:05 iamglass -rw------- 1 cyrus mail 13115392 Jul 21 15:05 position.glass -rw------- 1 cyrus mail 5545984 Jul 21 15:05 postlist.glass -rw------- 1 cyrus mail 3530752 Jul 21 15:05 termlist.glass I hope I can ignore this. I've found these errors only for one mailbox. == 2 == Another problem is that the "expire" annotation value is not configured as an "expire" metadata value on Cyrus 3.2.2. I don't know if this is expected or if this is a bug. But it is a problem for me, I have to rewrite all these mailboxes metadata. To be more clear, the Cyrus IMAP 2.4.20 annotation like * ANNOTATION "user/marco/Trash at example.com" "/vendor/cmu/cyrus-imapd/expire" ("value.shared" "14" "content-type.shared" "text/plain" "size.shared" "2" "modifiedsince.shared" "1406624279") become in Cyrus IMAP 3.2.2 after the sync: * METADATA user/marco/Trash at example.com ("/shared/vendor/cmu/cyrus-imapd/expire" NIL) I expected "14" and not NIL. == 3 == The specialuse flags are not kept in the replica server: * LIST (\HasNoChildren \Sent) "/" INBOX/Sent become * LIST (\HasNoChildren) "/" INBOX/Sent Do I have to suppose that I must run the cvt_xlist_specialuse tool as described in https://www.cyrusimap.org/3.2/imap/download/upgrade.html#upgrade-specific-items ? It's not clear to me if this instruction is related to the in place upgrade only or to the replication mode too. == 4 == This is a severe error. When I expunge a message in the master, the expunge action is not replicated on the replica, and the sync_client dies: 2020-07-23T14:50:44.695378+02:00 tst-msg01 cyrus/imap[26851]: Expunged 1 messages from example.com!user.marco^fff.Trash 2020-07-23T14:51:10.147600+02:00 tst-msg03 cyrus/replica2to3syncserver[1207 Fatal error: Internal error: assertion failed: imap/dlist.c: 156: base != NULL 2020-07-23T14:51:10.693261+02:00 tst-msg01 cyrus/sync_client[26834]: RESERVE received * response: 2020-07-23T14:51:10.693321+02:00 tst-msg01 cyrus/sync_client[26834]: reserve messages: failed: Bad protocol 2020-07-23T14:51:10.693730+02:00 tst-msg01 cyrus/sync_client[26834]: Error in do_sync(): bailing out! Bad protocol 2020-07-23T14:51:10.693750+02:00 tst-msg01 cyrus/sync_client[26834]: Processing sync log file /var/lib/imap/sync/replica2-3/log-26834 failed: Bad protocol 2020-07-23T14:51:10.706750+02:00 tst-msg03 cyrus/master[24009]: process type:SERVICE name:replica2to3syncserver path:/usr/libexec/cyrus-imapd/sync_server age:1142.993s pid:1207 exited, status 70 I can't no more suppress this error. All sync_client die, even in non rolling mode. I have to delete permanently the replicated mailboxes to resume the right working. I see the same error when I delete a mailbox on replica and after that I do a sync_client on master. In this last case the error disappear only when I delete the delayed DELETED mailbox too. I read the documentation, I didn't find restrictions on replica with the delayed expunged or the delayed deleted configuration in master and or in replica. In my environment I simply configured delayed deleted and delayed expunge in the same way, on master and on replica hosts. Could you help me to understand? Thank you very much Cheers Marco From g.danti at assyoma.it Fri Jul 31 07:39:23 2020 From: g.danti at assyoma.it (Gionatan Danti) Date: Fri, 31 Jul 2020 13:39:23 +0200 Subject: Restore message date from "Date:" field Message-ID: <3457db84ec8470540aea0d724ff76f80@assyoma.it> Hi all, I just noticed the dates of some old emails are wrongly displayed on roundcube webmail. In short, the list view shows the filesystem date of the affected messages (ie: mtime of u.1 file), rather than what is found in the "Date:" header field These were emails migrated from an old system, but I vaguely remember I had some issue at the time which I solved with some combination of rsync+imapsync. Can "reconstruct" be used to repopulate the index file with the correct date from "Date:" field? If not, what I can do to solve the issue? I already tried "reconstruct -u user at domain -x -f -r -G", but with no avail. Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From nic at nicbernstein.com Fri Jul 31 07:58:36 2020 From: nic at nicbernstein.com (Nic Bernstein) Date: Fri, 31 Jul 2020 06:58:36 -0500 Subject: Restore message date from "Date:" field In-Reply-To: <3457db84ec8470540aea0d724ff76f80@assyoma.it> References: <3457db84ec8470540aea0d724ff76f80@assyoma.it> Message-ID: <926a7dc0-cf4d-3dae-6ef3-e871c49d7bd3@nicbernstein.com> Gionatan, I believe the tool you're looking for is 'mbtool'? From the man page: DESCRIPTION mbtool is a tool for performing various actions on the indexes of a list of mailboxes. The only actions currently supported are -t, which will normalize the internaldate time stamp of each record in the index to GMT, and -r which will create a new unique ID for each mailbox. ... -t Normalize internaldate on all index records of all listed mail? boxes to match the Date: header if they?re off by more than a day, which can be used to fix up a mailbox which has been re? stored from backup and lost its internaldate information. ... EXAMPLES*mbtool -t* user.jsmith Normalize |internaldate| on all index records in /user.jsmith/. Working on user.jsmith... 00000001: Tue, 08 Jul 2014 16:45:18 -0500 => Mon, 07 Jul 2014 20:44:18 +0000 00000002: Tue Jul 08 16:45:13 CDT 2013 => Fri, 30 Aug 2013 19:46:03 +0000 <...> http://www.cyrusimap.org/imap/reference/manpages/systemcommands/mbtool.html?highlight=mbtool Cheers, ??? -nic On 7/31/20 6:39 AM, Gionatan Danti wrote: > Hi all, > I just noticed the dates of some old emails are wrongly displayed on > roundcube webmail. > > In short, the list view shows the filesystem date of the affected > messages (ie: mtime of u.1 file), rather than what is found in the > "Date:" header field > > These were emails migrated from an old system, but I vaguely remember > I had some issue at the time which I solved with some combination of > rsync+imapsync. > > Can "reconstruct" be used to repopulate the index file with the > correct date from "Date:" field? If not, what I can do to solve the > issue? I already tried "reconstruct -u user at domain -x -f -r -G", but > with no avail. > > Thanks. > -- Nic Bernstein nic at nicbernstein.com https://www.nicbernstein.com https://www.linkedin.com/in/nic-b-26577a178/ -------------- next part -------------- An HTML attachment was scrubbed... URL: