From ellie at fastmail.com Wed Jan 1 18:01:17 2020 From: ellie at fastmail.com (ellie timoney) Date: Thu, 02 Jan 2020 10:01:17 +1100 Subject: Migration issue with seen/subscription/sieve databases In-Reply-To: <3a9df4da960f672e02d5603563560c5c@assyoma.it> References: <291efd45-51cb-e9a8-cb6e-ade02594a397@assyoma.it> <1dc0f703-44c2-4077-74fa-02e1e197d6ae@assyoma.it> <81e2b832-9b8d-b470-d77e-6e275c7d8762@ednet.ns.ca> <3902c934-61cb-9057-0d3f-f461828dff4a@assyoma.it> <7f50db208e33fb0264a0d938aec0f8ee@assyoma.it> <7a69cf4f-646b-47e0-a788-d693e61285b5@www.fastmail.com> <3a9df4da960f672e02d5603563560c5c@assyoma.it> Message-ID: <54424d34-1abe-4014-9605-a5c77898638f@www.fastmail.com> On Mon, Dec 30, 2019, at 8:59 PM, Gionatan Danti wrote: > Are you referring to the problem described here [1]? If so, from the > linked page I read: > "Versions of 3.0 prior to 3.0.11 contained a bug (Issue #2839) that > could lead to loss of seen state/flags during reconstruct for some > messages that already existed prior to Cyrus 2.3" > > My Cyrus installation was never older than 2.3.x, so I thought the bug > should not affect me. Am I wrong? I think, in that case, you should be fine! It wasn't clear if your installation had only ever been 2.3, or had previously been upgraded to 2.3 from some earlier version in the past, so I thought I'd better point it out just in case :) Cheers, ellie From g.danti at assyoma.it Thu Jan 2 04:17:56 2020 From: g.danti at assyoma.it (Gionatan Danti) Date: Thu, 02 Jan 2020 10:17:56 +0100 Subject: Migration issue with seen/subscription/sieve databases In-Reply-To: <54424d34-1abe-4014-9605-a5c77898638f@www.fastmail.com> References: <291efd45-51cb-e9a8-cb6e-ade02594a397@assyoma.it> <1dc0f703-44c2-4077-74fa-02e1e197d6ae@assyoma.it> <81e2b832-9b8d-b470-d77e-6e275c7d8762@ednet.ns.ca> <3902c934-61cb-9057-0d3f-f461828dff4a@assyoma.it> <7f50db208e33fb0264a0d938aec0f8ee@assyoma.it> <7a69cf4f-646b-47e0-a788-d693e61285b5@www.fastmail.com> <3a9df4da960f672e02d5603563560c5c@assyoma.it> <54424d34-1abe-4014-9605-a5c77898638f@www.fastmail.com> Message-ID: Il 02-01-2020 00:01 ellie timoney ha scritto: > I think, in that case, you should be fine! It wasn't clear if your > installation had only ever been 2.3, or had previously been upgraded > to 2.3 from some earlier version in the past, so I thought I'd better > point it out just in case :) > > Cheers, > > ellie Hi Ellie, thanks again for your very useful comments. Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From cyrus at peralex.com Thu Jan 2 09:22:42 2020 From: cyrus at peralex.com (Mark) Date: Thu, 2 Jan 2020 16:22:42 +0200 Subject: Upgrade 2.4 -> 3: status 75, clean shutdown file missing Message-ID: Good day, I'm busy testing a change from Cyrus imapd version 2.4 -> 3.0.11 Everthing seems to be working, but there are a few things in the logs that I'm mildly concerned about when shutting down: imap[29733]: graceful shutdown imap[29732]: graceful shutdown imap[29724]: graceful shutdown idled[29725]: graceful shutdown imap[29735]: graceful shutdown master[29721]: process type:DAEMON name:idled path:/usr/local/cyrus/libexec/idled age:77.647s pid:29725 exited, status 75 master[29721]: process type:SERVICE name:imap path:/usr/local/cyrus/libexec/imapd age:4.806s pid:29735 exited, status 75 master[29721]: process type:SERVICE name:imap path:/usr/local/cyrus/libexec/imapd age:14.751s pid:29732 exited, status 75 master[29721]: process type:SERVICE name:imap path:/usr/local/cyrus/libexec/imapd age:5.437s pid:29734 exited, status 75 master[29721]: process type:SERVICE name:imap path:/usr/local/cyrus/libexec/imapd age:6.171s pid:29733 exited, status 75 master[29721]: process type:SERVICE name:imap path:/usr/local/cyrus/libexec/imapd age:77.648s pid:29724 exited, status 75 master[29721]: All children have exited, closing down In particular, is status 75 standard when shutting down? Then, when starting up again, I get a message about skiplist shutdown file being missing: ctl_cyrusdb[29750]: skiplist: clean shutdown file missing, updating recovery stamp Is this an issue?? Is it normal?? I'm not using skiplist for any of my databases: annotation_db: twoskip backup_db: twoskip conversations_db: twoskip duplicate_db: twoskip mboxkey_db: twoskip mboxlist_db: twoskip ptscache_db: twoskip quota_db: quotalegacy search_indexed_db: twoskip seenstate_db: twoskip sortcache_db: twoskip subscription_db: flat statuscache_db: twoskip tlscache_db: twoskip tls_sessions_db: twoskip userdeny_db: flat zoneinfo_db: twoskip Thanks, Mark Disclaimer: http://www2.peralex.com/disclaimer.html From gbulfon at sonicle.com Thu Jan 2 13:51:01 2020 From: gbulfon at sonicle.com (Gabriele Bulfon) Date: Thu, 2 Jan 2020 19:51:01 +0100 (CET) Subject: Squatter segfault on big mailbox In-Reply-To: <1399826115.779.1576563995688@www> References: <9279660e-7c86-a00d-45a5-e3516cafbf64@peralex.com> <1399826115.779.1576563995688@www> Message-ID: <1722971434.4532.1577991061957@www> Tried looking into the source of squatter and verbosing on a specific folder that always shows the problem. Here is how it ends: ? Processing index character 128, 9 total words, temp file size is 41 Processing index character 130, 8 total words, temp file size is 52 Processing index character 147, 2 total words, temp file size is 12 Processing index character 152, 2 total words, temp file size is 12 Processing index character 153, 5 total words, temp file size is 26 Processing index character 172, 10 total words, temp file size is 66 Processing index character 176, 72 total words, temp file size is 498 Processing index character 191, 417128205 total words, temp file size is 1251444163 Closing index: Not enough space ? I will look and change the source to give more info on the backtrace of the error, but maybe someone here alreay has any idea? ? Gabriele ? ? Sonicle S.r.l.? :? http://www.sonicle.com Music:? http://www.gabrielebulfon.com Quantum Mechanics :? http://www.cdbaby.com/cd/gabrielebulfon Da: Gabriele Bulfon A: Vladislav Kurz Mark info-cyrus Data: 17 dicembre 2019 7.26.35 CET Oggetto: Re: Squatter segfault on big mailbox ? Hello, we're having the same problem with Cyrus 2.5.11 on XStreamOS/illumos Any idea how to fix this? Same problem both on 32 bit large files build, and 64 bit. Looks like the Debian issue has no solution. Can't believe this has never been addressed. ? Gabriele ? ? Sonicle S.r.l.? :? http://www.sonicle.com Music:? http://www.gabrielebulfon.com Quantum Mechanics :? http://www.cdbaby.com/cd/gabrielebulfon ---------------------------------------------------------------------------------- Da: Vladislav Kurz A: Mark info-cyrus Data: 11 ottobre 2017 9.37.21 CEST Oggetto: Re: Squatter segfault on big mailbox On 10/11/17 07:37, Mark wrote: On 2017-10-10 16:05, Vladislav Kurz wrote: Hello everyone, we have recently migrated our mail server from cyrus 2.2 to 2.5 (both debian packages). We have a problem that squatter segfaults on some quite big mailboxes. I had to add -i (--incremental) to at least index new mails. But users cannot search in old mails (before migration). I used strace to find out if it is some particular mail, but it parsed everything fine, and failed when dealing with some of the many temporary files squatXXXX. The problematic mailboxes have just inbox10 GB (and15 GB with all subfolders) and20000 emails in inbox. Has anyone else hit similar problem and solved it somehow? I'm not sure about fixes, but there are some reports of similar problems.? Have a look at the thread with the subject "Cyrus 2.5.10 IMAP search" from the beginning of June this year.? Apparently the format of the squatter database changed between 2.4 and 2.5, and some people (at least) have seen a 10x increase in the size of the squatter database for the same mailbox.? For systems that only allow 2GB mmap, this can be a problem. I'm not sure if this increase in size is a bug or a feature. Mark Hello Mark, Thanks for the hint about old discussion. Unfortunately, there was no solution to this issue mentioned. I found out that this was already reported to debian, I wonder whetrher the maintainer forwarded the bug report upstream. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921 This is on 64-bit Linux, so mmap should not be limited to 2 GB. And I also have noticed the extreme increase in squat file sizes. -- Best Regards Vladislav Kurz ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus ----Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/To Unsubscribe:https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyrus at peralex.com Fri Jan 3 07:49:43 2020 From: cyrus at peralex.com (Mark) Date: Fri, 3 Jan 2020 14:49:43 +0200 Subject: Upgrade 2.4 -> 3: status 75, clean shutdown file missing In-Reply-To: References: Message-ID: <757b67c6-b806-cae0-a5f7-de96b195767d@peralex.com> On 2020-01-02 16:22, Mark wrote: > > > Then, when starting up again, I get a message about skiplist shutdown > file being missing: > > ctl_cyrusdb[29750]: skiplist: clean shutdown file missing, updating > recovery stamp > Digging into the code a bit, I find this (below) in cyrusdb_skiplist.c. Unless I'm mistaken, it seems that the file skipcleanshutdown is never actually created in the code, so this notice will always be generated. So I'd say that the 'clean shutdown file missing' message is benign. Am I correct that this file is never actually created? snprintf(cleanfile, sizeof(cleanfile), "%s/skipcleanshutdown", dbdir); /* if we had a clean shutdown, we don't need to run recovery on * everything */ if (stat(cleanfile, &sbuf) == 0) { syslog(LOG_NOTICE, "skiplist: clean shutdown detected, starting normally"); unlink(cleanfile); goto normal; } Change introduced here: https://github.com/cyrusimap/cyrus-imapd/commit/54f3cc4555f888072e3310aa481e51cba401b9fb#diff-d92d37e9b648c1bad26ee7f6e473bd5c Disclaimer: http://www2.peralex.com/disclaimer.html From roar.brenden.no at gmail.com Mon Jan 13 17:33:18 2020 From: roar.brenden.no at gmail.com (=?UTF-8?Q?Roar_Br=C3=A6nden?=) Date: Mon, 13 Jan 2020 23:33:18 +0100 Subject: Ftp server down Message-ID: Hi, I've tried to install a calendar server from calendarserver.org. That software is based on components from cyrusimap. During the setup there is a call for ftp.cyrusimap.org which fails with a "Connection timed out". It has been like this for a couple of days. Is this a known issue? Should the address be changed to something else? Best regards Roar Br?nden -------------- next part -------------- An HTML attachment was scrubbed... URL: From l.schimmer at cgv.tugraz.at Tue Jan 14 09:51:36 2020 From: l.schimmer at cgv.tugraz.at (Lars Schimmer) Date: Tue, 14 Jan 2020 15:51:36 +0100 Subject: Cyrus 3.0.8 and purging/removing emails Message-ID: <2fee4ada-f61c-7375-f855-31ae255ecb2b@cgv.tugraz.at> Hi! We do run a debian cyrus 3.0.8-6+deb10u3 mailserver, new setup, but we did copy over some coonfigs from a older cyrus. I recently discovered lots of emails in the local storage of the email server, although the mailbox is empty. In this case a IMAP shared folder, not a user INBOX. IMAP tells the folder is empty, but in the folder on the disk still 800 files/mails available, partly 3 month old. It seems I did miss the automatic removal of mails. The cyrus.conf shows the line delprune cmd="/usr/sbin/cyrus expire -E 3" But it seems I did miss some important point. On a cyrus restart it shows: cyrus/cyr_expire[13235]: Expired 0 and expunged 0 out of 1449266 messages from 3993 mailboxes Anyone has a idea what I did miss, or where to look? thank you. MfG, Lars Schimmer -- ------------------------------------------------------------- TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung Tel: +43 316 873-5405 E-Mail: l.schimmer at cgv.tugraz.at Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From fabio.montefuscolo at gmail.com Tue Jan 14 10:47:52 2020 From: fabio.montefuscolo at gmail.com (Fabio Montefuscolo) Date: Tue, 14 Jan 2020 12:47:52 -0300 Subject: Fwd: Help putting cyrus on Docker In-Reply-To: References: Message-ID: Hello! I'm trying to build a simple docker image based on CentOS 8, which brings cyrus-imapd 3.0.7. I'm having troubles to authenticate on cyrus imap service for unknown reasons. There is no syslog facility working inside the centos image, so I don't have logs. The final idea is deploy this image in a kubernetes cluster. The Dockerfile I'm using is https://github.com/fabiomontefuscolo/docker-cyrusimapd When I jump into console and try to use cyradm, I get 2 password fields to fulfill and at the ent, that doesn't work ``` {12:21}~/devel/cyrus:master ? ? docker exec -it cyrus bash [root at 256f72d885f9 /]# cyradm localhost --user cyrus Password: IMAP Password: Login failed: authentication failure at /usr/lib64/perl5/vendor_perl/Cyrus/IMAP/Admin.pm line 134. cyradm: cannot authenticate to server with as cyrus ``` Playing on telnet also does not work ``` {12:40}~/devel/cyrus:master ? ? telnet cyrus.docker 143 Trying 172.17.0.3... Connected to cyrus.docker. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=GS2-IAKERB AUTH=GS2-KRB5 AUTH=SCRAM-SHA-1 AUTH=SCRAM-SHA-256 AUTH=GSS-SPNEGO AUTH=GSSAPI AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=NTLM AUTH=PLAIN AUTH=LOGIN SASL-IR] 256f72d885f9 Cyrus IMAP 3.0.7-15.el8_0.1 Fedora server ready . login cyrus cyrus . NO Login failed: authentication failure ^] telnet> Connection closed. ``` Can someone give a clue on how get this working? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From karagian at gmail.com Tue Jan 14 10:51:49 2020 From: karagian at gmail.com (Savvas Karagiannidis) Date: Tue, 14 Jan 2020 17:51:49 +0200 Subject: Cyrus 3.0.8 and purging/removing emails In-Reply-To: <2fee4ada-f61c-7375-f855-31ae255ecb2b@cgv.tugraz.at> References: <2fee4ada-f61c-7375-f855-31ae255ecb2b@cgv.tugraz.at> Message-ID: Hi, according to the documentation for cyr_expire ( https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html) you also have to add -X parameter in order to completely delete messages from mailboxes and -D parameter in order to remove deleted mailboxes from the server. -E is only for expiring old entries in the duplicate messages databases. Note that the parameter "3" specified there is in days by default (3d). So, if you only want to keep data that was deleted the last, let's say, 7 days, and delete anything older than that, you enter the following in the EVENTS section of cyrus.conf: delprune cmd="/usr/sbin/cyrus/cyr_expire -E 3d -X 7d -D 7d" at 0400 The path to cyr_expire is not clear in your email, so please adjust it accordingly. "at 0400" specifies that the command must be executed daily at 04:00, you can of course change it. Regards, Savvas Karagiannidis On Tue, Jan 14, 2020, 16:52 Lars Schimmer wrote: > Hi! > > We do run a debian cyrus 3.0.8-6+deb10u3 mailserver, new setup, but we > did copy over some coonfigs from a older cyrus. > > I recently discovered lots of emails in the local storage of the email > server, although the mailbox is empty. > In this case a IMAP shared folder, not a user INBOX. > IMAP tells the folder is empty, but in the folder on the disk still 800 > files/mails available, partly 3 month old. > > It seems I did miss the automatic removal of mails. > > The cyrus.conf shows the line delprune cmd="/usr/sbin/cyrus > expire -E 3" > > But it seems I did miss some important point. > > On a cyrus restart it shows: > > cyrus/cyr_expire[13235]: Expired 0 and expunged 0 out of 1449266 > messages from 3993 mailboxes > > > Anyone has a idea what I did miss, or where to look? > > thank you. > > MfG, > Lars Schimmer > -- > ------------------------------------------------------------- > TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung > Tel: +43 316 873-5405 E-Mail: l.schimmer at cgv.tugraz.at > Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 > > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Tue Jan 14 20:16:12 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 15 Jan 2020 12:16:12 +1100 Subject: Ftp server down In-Reply-To: References: Message-ID: <2f1d11b6-3e04-4c76-9d27-da0c2bc58540@www.fastmail.com> Hi, That's a known issue, it's been down for months, and we don't really expect it to come back. All our releases are available over HTTPS from https://cyrusimap.org/releases/ Cheers, ellie On Tue, Jan 14, 2020, at 9:33 AM, Roar Br?nden wrote: > Hi, > > I've tried to install a calendar server from calendarserver.org. That software is based on components from cyrusimap. During the setup there is a call for ftp.cyrusimap.org which fails with a "Connection timed out". It has been like this for a couple of days. > > Is this a known issue? > > Should the address be changed to something else? > > Best regards > > Roar Br?nden > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From nd at syndicat.com Wed Jan 15 04:20:59 2020 From: nd at syndicat.com (Niels Dettenbach) Date: Wed, 15 Jan 2020 10:20:59 +0100 Subject: Fwd: Help putting cyrus on Docker In-Reply-To: References: Message-ID: <4746242.IOqI4mWz6t@gongo> Am Dienstag, 14. Januar 2020, 16:47:52 CET schrieb Fabio Montefuscolo: > I'm trying to build a simple docker image based on CentOS 8, which brings > cyrus-imapd 3.0.7. I'm having troubles to authenticate on cyrus imap > service for unknown reasons. There is no syslog facility working inside > the centos image, so I don't have logs. The final idea is deploy this > image in a kubernetes cluster. The Dockerfile I'm using is > https://github.com/fabiomontefuscolo/docker-cyrusimapd > > When I jump into console and try to use cyradm, I get 2 password fields to > fulfill and at the ent, that doesn't work as far as i can read in that dockerfile on a first view, it does only install dependencies and "activating" SSL/TLS, but lacks any kind of further required configuration for cyrus auth etc. the flexibility and complexity of cyrus installations are not easy to "capsule" into a docker file or reciept for a "common usage". depending on what kind of authentication subsystem (i.e. mysql, pam, ldap, pam_mysql, sasl, saslauthd (with pam or other "backend") you want to use or even active AD or others you have to configure that by hand (or extend that docker file). a typical "easy" way is using saslauthd with -s pam to "simply" use pam authentication. hth, niels. -- --- Niels Dettenbach Syndicat IT & Internet http://www.syndicat.com PGP: https://syndicat.com/pub_key.asc --- From fabio.montefuscolo at gmail.com Wed Jan 15 06:49:46 2020 From: fabio.montefuscolo at gmail.com (Fabio Montefuscolo) Date: Wed, 15 Jan 2020 08:49:46 -0300 Subject: Fwd: Help putting cyrus on Docker In-Reply-To: <4746242.IOqI4mWz6t@gongo> References: <4746242.IOqI4mWz6t@gongo> Message-ID: Hi Niels Thank you very much for looking into this. Initially, I would like to get this working like I got in a real CentOS 8. I have here a virtual machine where I did the same steps I did in Dockerfile. But in VM I start services "*systemctl start cyrus-imapd*" and "*systemctl start saslauthd*" and basic login works out of the box. I could telnet on port 143 and "*. login cyrus cyrus*". That is not happening on Docker. I hope other people using the image can write their own config and mount inside the container to achieve their needs. The plan is to have some kubernetes recipes I can repeat whenever we get a new client wondering to have an email service. Actually I'm testing it on DO, but using kubernetes should be simple to move to AWS, Linode or any other kubernetes hosting. Thank you Em qua., 15 de jan. de 2020 ?s 06:22, Niels Dettenbach via Info-cyrus < info-cyrus at lists.andrew.cmu.edu> escreveu: > Am Dienstag, 14. Januar 2020, 16:47:52 CET schrieb Fabio Montefuscolo: > > I'm trying to build a simple docker image based on CentOS 8, which brings > > cyrus-imapd 3.0.7. I'm having troubles to authenticate on cyrus imap > > service for unknown reasons. There is no syslog facility working inside > > the centos image, so I don't have logs. The final idea is deploy this > > image in a kubernetes cluster. The Dockerfile I'm using is > > https://github.com/fabiomontefuscolo/docker-cyrusimapd > > > > When I jump into console and try to use cyradm, I get 2 password fields > to > > fulfill and at the ent, that doesn't work > > > as far as i can read in that dockerfile on a first view, it does only > install > dependencies and "activating" SSL/TLS, but lacks any kind of further > required > configuration for cyrus auth etc. > > the flexibility and complexity of cyrus installations are not easy to > "capsule" into a docker file or reciept for a "common usage". > > depending on what kind of authentication subsystem (i.e. mysql, pam, ldap, > pam_mysql, sasl, saslauthd (with pam or other "backend") you want to use > or > even active AD or others you have to configure that by hand (or extend > that > docker file). > > a typical "easy" way is using saslauthd with -s pam to "simply" use pam > authentication. > > > hth, > > > > niels. > > > -- > --- > Niels Dettenbach > Syndicat IT & Internet > http://www.syndicat.com > PGP: https://syndicat.com/pub_key.asc > --- > > > > > > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l.schimmer at cgv.tugraz.at Wed Jan 15 10:21:03 2020 From: l.schimmer at cgv.tugraz.at (Lars Schimmer) Date: Wed, 15 Jan 2020 16:21:03 +0100 Subject: Cyrus 3.0.8 and purging/removing emails In-Reply-To: References: <2fee4ada-f61c-7375-f855-31ae255ecb2b@cgv.tugraz.at> Message-ID: <63f37f96-e3ea-0f22-3cad-fcda83b2be37@cgv.tugraz.at> On 2020-01-14 16:51, Savvas Karagiannidis wrote: > Hi, > according to the documentation for cyr_expire > (https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html) > you also have to add -X parameter in order to completely delete messages > from mailboxes and -D parameter in order to remove deleted mailboxes > from the server. -E is only for expiring old entries in the duplicate > messages databases. Note that the parameter "3" specified there is in > days by default (3d). > > So, if you only want to keep data that was deleted the last, let's say, > 7 days, and delete anything older than that, you enter the following in > the EVENTS section of cyrus.conf: > > delprune? ? ? ? cmd="/usr/sbin/cyrus/cyr_expire -E 3d -X 7d -D 7d" at 0400 > > The path to cyr_expire is not clear in your email, so please adjust it > accordingly. > "at 0400" specifies that the command must be executed daily at 04:00, > you can of course change it. Thank you! This solved the issue. > Regards, > Savvas Karagiannidis MfG, Lars Schimmer -- ------------------------------------------------------------- TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung Tel: +43 316 873-5405 E-Mail: l.schimmer at cgv.tugraz.at Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From hh at promedia-sds.de Fri Jan 17 03:52:03 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 09:52:03 +0100 Subject: Confused about Deleted and Expunged Message-ID: <46fd8ba0-ad4e-32ac-694d-9604a7ce4a9d@promedia-sds.de> Hi, my Mail Server stores spam mail via spamassassin in users (user/test at testdomain.tld) spam mailboxes (user/test/spam at testdomain.tld) For removing too old spam mails, i do use a script on all mailboxes which marks the spam mailboxes as deleted: /usr/lib/cyrus/bin/ipurge -f -X -d 100 user/test/spam at testdomain.tld This works perfect, the user can see spam mails for 100 days. I assumed the following cyrus.conf event does Expunge the deleted mails from hdd: [...] EVENTS { expungeprune cmd="/usr/sbin/cyrus expire -E 4 -D 28 -X 28" at=0445 [...] But this does not work. How can i 'really' delete from hdd?? thanks for your help Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From Hagedorn at uni-koeln.de Fri Jan 17 04:24:09 2020 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 17 Jan 2020 10:24:09 +0100 Subject: Confused about Deleted and Expunged In-Reply-To: <46fd8ba0-ad4e-32ac-694d-9604a7ce4a9d@promedia-sds.de> References: <46fd8ba0-ad4e-32ac-694d-9604a7ce4a9d@promedia-sds.de> Message-ID: Hi, Am 17.01.20 um 09:52 Uhr schrieb Horst H?berlen: > my Mail Server stores spam mail via spamassassin in users (user/test at testdomain.tld) spam > mailboxes (user/test/spam at testdomain.tld) > > For removing too old spam mails, i do use a script on all mailboxes which marks the spam > mailboxes as deleted: > /usr/lib/cyrus/bin/ipurge -f -X -d 100 user/test/spam at testdomain.tld > > This works perfect, the user can see spam mails for 100 days. > > I assumed the following cyrus.conf event does Expunge the deleted mails from hdd: > [...] > > EVENTS { > expungeprune cmd="/usr/sbin/cyrus expire -E 4 -D 28 -X 28" at=0445 > > [...] > > But this does not work. > > How can i 'really' delete from hdd?? does it really say "/usr/sbin/cyrus expire" in your cyrus.conf? That can't work. The command is named cyr_expire on my system, and it definitely does not have a space in its name. -------------- next part -------------- A non-text attachment was scrubbed... Name: Hagedorn.vcf Type: text/x-vcard Size: 333 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5367 bytes Desc: S/MIME Cryptographic Signature URL: From hh at promedia-sds.de Fri Jan 17 05:11:09 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 11:11:09 +0100 Subject: Confused about Deleted and Expunged Message-ID: <509515f0-0428-a0cd-26d5-c86f568363ff@promedia-sds.de> Hi Sebastian Hagedorn , i have no idea how follow-up works in this list, i hope this mail goes to the right place. > does it really say "/usr/sbin/cyrus expire" in your cyrus.conf? That > can't work. The command is named cyr_expire on my system, and it > definitely does not have a space in its name. Yes, the command is correct. On Ubuntu 18.04 there is no cyr_expire and the entry "/usr/sbin/cyrus expire" is in cyrus.conf by default. In the log files the results are printed ".. localhost cyrus/cyr_expire[14320]: Expired 0 and expunged 0 out of 143844 messages from 818 mailboxes" regards Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From ngatsis at qbit.gr Fri Jan 17 07:11:52 2020 From: ngatsis at qbit.gr (Nikos Gatsis - Qbit) Date: Fri, 17 Jan 2020 14:11:52 +0200 Subject: Confused about Deleted and Expunged In-Reply-To: <509515f0-0428-a0cd-26d5-c86f568363ff@promedia-sds.de> References: <509515f0-0428-a0cd-26d5-c86f568363ff@promedia-sds.de> Message-ID: An HTML attachment was scrubbed... URL: From boutilpj at ednet.ns.ca Fri Jan 17 07:17:56 2020 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 17 Jan 2020 08:17:56 -0400 Subject: Confused about Deleted and Expunged In-Reply-To: References: <509515f0-0428-a0cd-26d5-c86f568363ff@promedia-sds.de> Message-ID: Looks to run cyr_expire as /usr/sbin/cyrus is a script. https://www.apt-browse.org/browse/ubuntu/trusty/universe/all/cyrus-common/2.4.17+caldav~beta9-3/file/usr/sbin/cyrus On 1/17/20 8:11 AM, Nikos Gatsis - Qbit wrote: > delprune? cmd="cyr_expire -E 3" at=0400 > > ?n centos 7: > > Version: 2.4.17 > > Release: 15.el7 > > > > On 17/1/2020 12:11 ?.?., Horst H?berlen wrote: >> Hi Sebastian Hagedorn , >> >> i have no idea how follow-up works in this list, i hope this mail goes to the right place. >> >>> does it really say "/usr/sbin/cyrus expire" in your cyrus.conf? That >>> can't work. The command is named cyr_expire on my system, and it >>> definitely does not have a space in its name. >> Yes, the command is correct. On Ubuntu 18.04 there is no cyr_expire and the entry >> "/usr/sbin/cyrus expire" is in cyrus.conf by default. >> >> In the log files the results are printed ".. localhost cyrus/cyr_expire[14320]: Expired 0 >> and expunged 0 out of 143844 messages from 818 mailboxes" >> >> regards >> Horst >> >> > -- > Untitled Document > ------------------------------------------------------------------------ > *?????? ????? - Gatsis Nikos* > Web developer > tel.: 210.8256721 - 210.8256712 > email: ngatsis at qbit.gr > http://www.qbit.gr > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From hh at promedia-sds.de Fri Jan 17 07:35:58 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 13:35:58 +0100 Subject: FWD: Confused about Deleted and Expunged Message-ID: <73923afd-5ac4-5427-494b-fb7ebe197f36@promedia-sds.de> Hi Nikos, > does it really say "/usr/sbin/cyrus expire" in your cyrus.conf? That > can't work. The command is named cyr_expire on my system, and it > definitely does not have a space in its name. YES there is no space in the name, it seems to be a parameter for cyrus. This command seems to work on Ubuntu 18.04. (cyrus-imapd 2.5.10-3ubuntu1) In /usr/sbin there is no cyr_expire but now i found a /usr/lib/cyrus/bin/cyr_expire The result is the same, no files were deleted, the log file entry is the same as when i use the "/usr/sbin/cyrus expire [..]" command Can anybody confirm this "/usr/sbin/cyrus expire [..]" command on Ubuntu? Regards Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From awilliam at whitemice.org Fri Jan 17 08:25:52 2020 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Fri, 17 Jan 2020 08:25:52 -0500 Subject: FWD: Confused about Deleted and Expunged In-Reply-To: <73923afd-5ac4-5427-494b-fb7ebe197f36@promedia-sds.de> References: <73923afd-5ac4-5427-494b-fb7ebe197f36@promedia-sds.de> Message-ID: <1579267552.4676.0.camel@whitemice.org> > This command seems to work on Ubuntu 18.04. (cyrus-imapd 2.5.10- > 3ubuntu1) > In /usr/sbin there is no cyr_expire but now i found a > /usr/lib/cyrus/bin/cyr_expire > The result is the same, no files were deleted, the log file entry is > the same as when i > use the "/usr/sbin/cyrus expire [..]" command > Can anybody confirm this "/usr/sbin/cyrus expire [..]" command on > Ubuntu? What happens if you su to the cyrus user and try to run the command you see in cyrus.conf ? -- Adam Tauno Williams, awilliam at whitemice.org Multi-Modal Activists Against Auto Dependent Development resisting the unAmerican socialists of the Motorist hegemony http://www.mmaaadd.org From michael.menge at zdv.uni-tuebingen.de Fri Jan 17 09:36:38 2020 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Fri, 17 Jan 2020 15:36:38 +0100 Subject: Confused about Deleted and Expunged In-Reply-To: <1579267552.4676.0.camel@whitemice.org> References: <73923afd-5ac4-5427-494b-fb7ebe197f36@promedia-sds.de> <1579267552.4676.0.camel@whitemice.org> Message-ID: <20200117153638.Horde.h5aMMGDmCbIEIpIugQvFiCB@webmail.uni-tuebingen.de> Hi I suspect a confusion about "marked as Deleted", Expunged and Expired See https://www.cyrusimap.org/imap/reference/faqs/o-deleted-expired-expunged-purged.html of more information. E-Mails "marked as Deleted" need to be Expunged by the imap EXPUNGE or UID EXPUNGE command. The cyr_expire cmd is for removing Expunged mails from the index and disk if "expunge_mode: delayed" is used, which is the default since 2.5 You may want to take a look at the "/vendor/cmu/cyrus-imapd/expire" mailbox annotation Quoting Adam Tauno Williams : >> This command seems to work on Ubuntu 18.04. (cyrus-imapd 2.5.10- >> 3ubuntu1) >> In /usr/sbin there is no cyr_expire but now i found a >> /usr/lib/cyrus/bin/cyr_expire >> The result is the same, no files were deleted, the log file entry is >> the same as when i >> use the "/usr/sbin/cyrus expire [..]" command >> Can anybody confirm this "/usr/sbin/cyrus expire [..]" command on >> Ubuntu? > > What happens if you su to the cyrus user and try to run the command you > see in cyrus.conf ? > > -- > Adam Tauno Williams, awilliam at whitemice.org > Multi-Modal Activists Against Auto Dependent Development > resisting the unAmerican socialists of the Motorist hegemony > http://www.mmaaadd.org > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From hh at promedia-sds.de Fri Jan 17 09:41:36 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 15:41:36 +0100 Subject: FWD: Confused about Deleted and Expunged Message-ID: hi Adam Tauno Williams, > What happens if you su to the cyrus user and try to run the command you > see in cyrus.conf ? This makes no difference. I tried both commands as root and as cyrus. The Log entries were identical: [...] cyrus/cyr_expire[10175]: Expired 0 and expunged 0 out of 144646 messages from 818 mailboxes [...] cyrus/cyr_expire[10175]: Removed 0 deleted mailboxes [...] cyrus/cyr_expire[10175]: duplicate_prune: pruning back 4.00 days [...] cyrus/cyr_expire[10175]: duplicate_prune: purged 0 out of 4171 entries regards Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From hh at promedia-sds.de Fri Jan 17 10:02:15 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 16:02:15 +0100 Subject: Confused about Deleted and Expunged Message-ID: > Looks to run cyr_expire as /usr/sbin/cyrus is a script. > > https://www.apt-browse.org/browse/ubuntu/trusty/universe/all/cyrus-common/2.4.17+caldav~beta9-3/file/usr/sbin/cyrus Thanks for the clarification! -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From boutilpj at ednet.ns.ca Fri Jan 17 10:04:48 2020 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 17 Jan 2020 11:04:48 -0400 Subject: FWD: Confused about Deleted and Expunged In-Reply-To: References: Message-ID: <14143db2-0b1b-9c40-e557-488edab9416d@ednet.ns.ca> Anything in the logs showing email being expunged? Look for Expunged with the capitol E. On 1/17/20 10:41 AM, Horst H?berlen wrote: > hi Adam Tauno Williams, > >> What happens if you su to the cyrus user and try to run the command you >> see in cyrus.conf ? > > This makes no difference. I tried both commands as root and as cyrus. The Log entries were > identical: > > [...] cyrus/cyr_expire[10175]: Expired 0 and expunged 0 out of 144646 messages from 818 > mailboxes > [...] cyrus/cyr_expire[10175]: Removed 0 deleted mailboxes > [...] cyrus/cyr_expire[10175]: duplicate_prune: pruning back 4.00 days > [...] cyrus/cyr_expire[10175]: duplicate_prune: purged 0 out of 4171 entries > > > regards > Horst > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 307 bytes Desc: not available URL: From hh at promedia-sds.de Fri Jan 17 10:13:41 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 16:13:41 +0100 Subject: Confused about Deleted and Expunged Message-ID: Hi M.Menge , > I suspect a confusion about "marked as Deleted", Expunged and Expired > > See > https://www.cyrusimap.org/imap/reference/faqs/o-deleted-expired-expunged-purged.html > of more information. > > E-Mails "marked as Deleted" need to be Expunged by the imap > EXPUNGE or UID EXPUNGE command. > > The cyr_expire cmd is for removing Expunged mails from the index and disk > if "expunge_mode: delayed" is used, which is the default since 2.5 > > You may want to take a look at the "/vendor/cmu/cyrus-imapd/expire" > mailbox annotation My command "/usr/lib/cyrus/bin/ipurge -f -X -d 100 user/test/spam at testdomain.tld" gives me: Working on user/test/spam at testdomain.tld... Total messages 9214 Total bytes 191140555 Deleted messages 1 Deleted bytes 2250912 Remaining messages 9213 Remaining bytes 188889643 So i see it markin as /Deleted works. Access by imap client gives the number 9213. But in the filesystem there are many more files in the spam folder: 9213 which can be 'seen' and many more which are marked as /Deleted. The command "/usr/sbin/cyrus expire -E 4 -D 28 -X 28" should Expunge (which is, what i suppose, removing from filesystem) the /Deleted marked files. Am i wrong? regards Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From hh at promedia-sds.de Fri Jan 17 10:33:22 2020 From: hh at promedia-sds.de (=?UTF-8?Q?Horst_H=c3=a4berlen?=) Date: Fri, 17 Jan 2020 16:33:22 +0100 Subject: FWD: Confused about Deleted and Expunged Message-ID: Hi Patrick, > Anything in the logs showing email being expunged? Look for Expunged > with the capitol E. I found some entries like: [...] cyrus/imap[5918]: Expunged 1 messages from test.tld!user.test.spam What is the difference between the above entry and the below entry (lower and capital expunged)? [...] cyrus/cyr_expire[14039]: Expired 0 and expunged 0 out of 144674 messages from 818 mailboxes The "Expunged" message should come from some imap-client (don't know which)? The "expunged" are from the command line execution of "/usr/sbin/cyrus expire [...]". regards Horst -- Horst H?berlen Kirchenstr. 16 74639 Zweiflingen From awilliam at whitemice.org Fri Jan 17 10:53:20 2020 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Fri, 17 Jan 2020 10:53:20 -0500 Subject: FWD: Confused about Deleted and Expunged In-Reply-To: References: Message-ID: <1579276400.4676.6.camel@whitemice.org> > I found some entries like: > [...] cyrus/imap[5918]: Expunged 1 messages from > test.tld!user.test.spam > What is the difference between the above entry and the below entry > (lower and capital > expunged)? > [...] cyrus/cyr_expire[14039]: Expired 0 and expunged 0 out of 144674 > messages from 818 > mailboxes > The "Expunged" message should come from some imap-client (don't know > which)? Yes, an IMAP client issued an EXPUNGE. > The "expunged" are from the command line execution of > "/usr/sbin/cyrus expire [...]". This looks for Expunged messages to Expire. It is a three phased delete; which is beautiful. Message Deleted (becomes candidate for expunge) -> Message Expunged (becomes candidate for expiration) -> Message Expired The first two steps are performed by the client/user. The final step is performed by the administrator, expressed via the server's policy as defined in cyrus.conf. From daniel at gultsch.de Thu Jan 23 06:34:14 2020 From: daniel at gultsch.de (Daniel Gultsch) Date: Thu, 23 Jan 2020 11:34:14 +0000 Subject: Unable to build Cyrus Message-ID: Hi, I?m (suddenly) unable to build Cyrus. I've been building the master branch in irregular intervals over the last year or so. My last successful build was on December 28th; but after that I haven?t been able to. The system libraries haven?t changed since then. I?m getting the following error: /usr/bin/ld: warning: libicui18n.so.57, needed by /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, may conflict with libicui18n.so.64 /usr/bin/ld: warning: libicuuc.so.57, needed by /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, may conflict with libicuuc.so.64 /usr/bin/ld: warning: libicudata.so.57, needed by /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, may conflict with libicudata.so.64 imap/ctl_zoneinfo.o: In function `main': /root/cyrus-imapd/imap/ctl_zoneinfo.c:232: undefined reference to `xmlGetNextNode' /root/cyrus-imapd/imap/ctl_zoneinfo.c:234: undefined reference to `xmlGetNextNode' /root/cyrus-imapd/imap/ctl_zoneinfo.c:240: undefined reference to `xmlGetNextNode' /root/cyrus-imapd/imap/ctl_zoneinfo.c:251: undefined reference to `xmlGetNextNode' /root/cyrus-imapd/imap/ctl_zoneinfo.c:253: undefined reference to `xmlGetNextNode' collect2: error: ld returned 1 exit status Makefile:3639: recipe for target 'imap/ctl_zoneinfo' failed This is a bit weird to me since according to git; the ctl_zoneinfo.c file hasn?t changed either. I?m using these commands to build Cyrus: export CYRUSLIBS=/usr/local/cyruslibs export PKG_CONFIG_PATH="$CYRUSLIBS/lib/pkgconfig:$PKG_CONFIG_PATH" export LDFLAGS="-Wl,-rpath,$CYRUSLIBS/lib -Wl,-rpath,$CYRUSLIBS/lib/x86_64-linux-gnu" export PATH="$CYRUSLIBS/bin:$CYRUSLIBS/sbin:$CYRUSLIBS/libexec:$PATH" cd cyrus-imapd make distclean git clean -f -x -d autoreconf -v -i automake -v -i ./configure --enable-jmap --enable-xapian --enable-http XAPIAN_CONFIG="$CYRUSLIBS/bin/xapian-config-1.5" make I?m happy for any hints on how to resolve that. cheers Daniel From cyrus at geoff.dj Thu Jan 23 07:02:04 2020 From: cyrus at geoff.dj (Geoff Winkless) Date: Thu, 23 Jan 2020 12:02:04 +0000 Subject: Unable to build Cyrus In-Reply-To: References: Message-ID: On Thu, 23 Jan 2020 at 11:34, Daniel Gultsch wrote: > /usr/bin/ld: warning: libicui18n.so.57, needed by > /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, > may conflict with libicui18n.so.64 At first guess, maybe update libxml (or libxml-devel) package? It looks like you've updated the i18n package but not the libxml one? Geoff From egoitz at sarenet.es Thu Jan 23 12:38:22 2020 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Thu, 23 Jan 2020 18:38:22 +0100 Subject: Question about squatter for Xapian Message-ID: <29E7AB11-6E70-4A55-B9E5-E6055F32FD10@sarenet.es> Good morning, We have mailboxes indexed with Xapian. We run for that Squatter in rolling mode (-R) so that a log can create new index records while new mail enters.. is removed and so? Nightly too move records between tiers and so? In an emergency on one mailbox server, we moved traffic to what was the slave server, and from that moment obviously the master? but we did a mistake, we forgot setting in the new slave "sync_log: true" so no squatter rolling log was generated. We noted about it 12 hours later. As we have "search_fuzzy_always: 1? set all searches go through Xapian, so no indexed mail won?t never appear in the searchs run by the users. I think we could run squatter for indexing non indexed emails, but I was wondering : - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I assume it should be extremely slow? so could this be launched?. Could you advise me please, if another way is preferred? - I assume not, but as we move records between Xapian tiers nightly? if the Squatter launched by me, by hand (for those non indexed emails), runs at the same time as this between tiers movement of records or at the same time too as the rolling mode squatter (-R) could one squatter process interfere in the job of the other instance of squatter?. Thank you so much, Best regards, From robn at fastmail.com Thu Jan 23 17:04:54 2020 From: robn at fastmail.com (=?UTF-8?Q?Rob_N_=E2=98=85?=) Date: Fri, 24 Jan 2020 09:04:54 +1100 Subject: Question about squatter for Xapian In-Reply-To: <29E7AB11-6E70-4A55-B9E5-E6055F32FD10@sarenet.es> References: <29E7AB11-6E70-4A55-B9E5-E6055F32FD10@sarenet.es> Message-ID: <33b2d39c-6d69-4e63-9c8a-a5a9f777f64a@www.fastmail.com> On Fri, 24 Jan 2020, at 4:38 AM, egoitz at sarenet.es wrote: > - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I assume it should be extremely slow? so could this be launched?. Could you advise me please, if another way is preferred? Normally, just the non-indexed emails. squatter -i (incremental) should be all you need to fill the gaps in your index. Obviously how long it takes depends on how much mail has arrived and how good your disks are, but for 12 hours worth I wouldn't expect more than a couple of hours to fill the gaps. > - I assume not, but as we move records between Xapian tiers nightly? if the Squatter launched by me, by hand (for those non indexed emails), runs at the same time as this between tiers movement of records or at the same time too as the rolling mode squatter (-R) could one squatter process interfere in the job of the other instance of squatter?. It's ok to run them all at the same time. Cyrus has appropriate locks to make sure that Xapian updates and repacks don't get in each others' way. Rob N. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Thu Jan 23 18:31:38 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 24 Jan 2020 10:31:38 +1100 Subject: Unable to build Cyrus In-Reply-To: References: Message-ID: <04ea0c60-81dc-461f-a9dc-2f1b3095a5d6@www.fastmail.com> Hi Daniel, On Thu, Jan 23, 2020, at 10:34 PM, Daniel Gultsch wrote: > /usr/bin/ld: warning: libicui18n.so.57, needed by > /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, > may conflict with libicui18n.so.64 > /usr/bin/ld: warning: libicuuc.so.57, needed by > /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, > may conflict with libicuuc.so.64 > /usr/bin/ld: warning: libicudata.so.57, needed by > /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so, > may conflict with libicudata.so.64 For starters, I would ignore these warnings for now, unless it turns out to be an actual problem. What's happening here is you have libicu v64 installed via cyruslibs, and you have libxml installed from a system package. libxml depends on libicu, which means you also have the system package of libicu, which in this case is v57. (I guess you're on something akin to Debian Jessie, cause I am, and I also have libicu v57 installed for libxml.) You will see these warnings if the link order is not quite right, or if the link order is correct but the linker is linking stuff it doesn't need to. It would be interesting to see what happens if you add "-Wl,--as-needed" to your LDFLAGS -- this is very useful for shaking out link ordering problems -- but I don't (yet) think that's the cause of the problem below. Some platforms have this linker option enabled by default, but Debian Jessie doesn't. > imap/ctl_zoneinfo.o: In function `main': > /root/cyrus-imapd/imap/ctl_zoneinfo.c:232: undefined reference to > `xmlGetNextNode' > /root/cyrus-imapd/imap/ctl_zoneinfo.c:234: undefined reference to > `xmlGetNextNode' > /root/cyrus-imapd/imap/ctl_zoneinfo.c:240: undefined reference to > `xmlGetNextNode' > /root/cyrus-imapd/imap/ctl_zoneinfo.c:251: undefined reference to > `xmlGetNextNode' > /root/cyrus-imapd/imap/ctl_zoneinfo.c:253: undefined reference to > `xmlGetNextNode' > collect2: error: ld returned 1 exit status > Makefile:3639: recipe for target 'imap/ctl_zoneinfo' failed This is where things get interesting. xmlGetNextNode is a fallback we provide for when libxml doesn't provide xmlFirstElementChild (in which case we #define xmlFirstElementChild to call our own xmlGetNextNode). So it sounds like configure might be confused about whether your libxml provides xmlFirstElementChild or not... Do you happen to have multiple versions of libxml installed, I wonder? If you remove the following files from your tree: ./imap/ctl_zoneinfo.o ./imap/ctl_zoneinfo ./imap/.deps/ctl_zoneinfo.Po ./imap/.libs/ctl_zoneinfo (i.e. the generated files for the ctl_zoneinfo program) and then run "make V=1 imap/ctl_zoneinfo" to rebuild just this program, what commands are being run to rebuild this file? The V=1 ensures the full commands are output, overriding --enable-silent-rules. The contents of your config.h and config.log would also be very helpful, but please check/sanitise them carefully for private information before attaching them. Feel free to send them directly to me (ellie at fastmail dot com) if you'd rather not share them with the whole list. Cheers, ellie From egoitz at ramattack.net Fri Jan 24 02:14:50 2020 From: egoitz at ramattack.net (Egoitz Aurrekoetxea) Date: Fri, 24 Jan 2020 08:14:50 +0100 Subject: Question about squatter for Xapian In-Reply-To: <33b2d39c-6d69-4e63-9c8a-a5a9f777f64a@www.fastmail.com> References: <29E7AB11-6E70-4A55-B9E5-E6055F32FD10@sarenet.es> <33b2d39c-6d69-4e63-9c8a-a5a9f777f64a@www.fastmail.com> Message-ID: <1B005685-1D57-49F9-8519-EC64653A94CC@ramattack.net> Hi!! Thank you so much Rob!! I will launch it this weekend :) :) Cheers! > El 23 ene 2020, a las 23:04, Rob N ? escribi?: > > On Fri, 24 Jan 2020, at 4:38 AM, egoitz at sarenet.es wrote: >> - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I assume it should be extremely slow? so could this be launched?. Could you advise me please, if another way is preferred? > > Normally, just the non-indexed emails. > > squatter -i (incremental) should be all you need to fill the gaps in your index. > > Obviously how long it takes depends on how much mail has arrived and how good your disks are, but for 12 hours worth I wouldn't expect more than a couple of hours to fill the gaps. > >> - I assume not, but as we move records between Xapian tiers nightly? if the Squatter launched by me, by hand (for those non indexed emails), runs at the same time as this between tiers movement of records or at the same time too as the rolling mode squatter (-R) could one squatter process interfere in the job of the other instance of squatter?. > > It's ok to run them all at the same time. Cyrus has appropriate locks to make sure that Xapian updates and repacks don't get in each others' way. > > Rob N. > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at ramattack.net Mon Jan 27 03:51:19 2020 From: egoitz at ramattack.net (Egoitz Aurrekoetxea) Date: Mon, 27 Jan 2020 09:51:19 +0100 Subject: Database upgrade and Xapian version dependency Message-ID: Hi!, Just for having it slightly clearer? When you upgrade the Cyrus version and the version you are upgrading to is a too close one? for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both? is it needed to launch (or recommended for some reason) the final upgrade commands : reconstruct -V max ctl_conversationsdb -b -r quota -f Or perhaps in the release notes should be seen they are needed due to a database upgrade possibility?. By the way, does exist any kind of Xapian needed version for the last 3.0.13 version?. I?m running Xapian 1.4.9, it?s pretty new? Thanks a lot mates! Bye! From zorg at probesys.com Tue Jan 28 04:22:43 2020 From: zorg at probesys.com (Zorg) Date: Tue, 28 Jan 2020 10:22:43 +0100 Subject: Create ics in another calendar (internal server error) Message-ID: <9cc1bd8f-ed6c-60e8-f0ec-28e2364b1f5b@probesys.com> Hi Let me explain I want my admin to write event in default calendar of my user For example myadmin at my.domain.com need to create event in user/myuser1/#calendars/Default put every time i this to add event (thunderbird, evolution, curl ) i have this in the log? "HTTP/1.1 500 Internal Server Error" (error=The server encountered an internal error.) What am i missing ? Here is the mailbox i have and acl user/myuser1/#calendars/Attachments at my.domain.com (\HasNoChildren) user/myuser1/#calendars/Default at my.domain.com (\HasNoChildren) user/myuser1/#calendars/Inbox at my.domain.com (\HasNoChildren) user/myuser1/#calendars/Outbox at my.domain.com (\HasNoChildren) user/myuser1/#calendars at my.domain.com (\HasNoChildren) user/myuser1/#notifications at my.domain.com (\HasNoChildren) user/myuser1 at my.domain.com (\HasNoChildren) user/myuser1 at my.domain.com: ? myadmin at my.domain.com lrswipkxtecdan ? myuser1 at my.domain.com lrswipkxtecdan user/myuser1/#calendars at my.domain.com: ? anyone 9 ? myadmin at my.domain.com lrswipkxtecdan9 ? myuser1 at my.domain.com lrswipkxtecdan9 user/myuser1/#calendars/Attachments at my.domain.com: ? myuser1 at my.domain.com lrswipkxtecdan ? myadmin at my.domain.com lrswipkxtecdan9 ? anyone r user/myuser1/#calendars/Default at my.domain.com: ? anyone 9 ? myadmin at my.domain.com lrswipkxtecdan9 ? myuser1 at my.domain.com lrswipkxtecdan9 user/myuser1/#calendars/Inbox at my.domain.com: ? myadmin at my.domain.com lrswipkxtecdan9 ? anyone 789 ? myuser1 at my.domain.com lrswipkxtecdan789 user/myuser1/#calendars/Outbox at my.domain.com: ? myadmin at my.domain.com lrswipkxtecdan9 ? anyone 9 ? myuser1 at my.domain.com lrswipkxtecdan789 user/myuser1/#notifications at my.domain.com: ? myuser1 at my.domain.com lrswipkxtecdan ? myadmin at my.domain.com lrswipkxtecdan9 Thanks From rsto at fastmailteam.com Tue Jan 28 04:26:26 2020 From: rsto at fastmailteam.com (Robert Stepanek) Date: Tue, 28 Jan 2020 10:26:26 +0100 Subject: Create ics in another calendar (internal server error) In-Reply-To: <9cc1bd8f-ed6c-60e8-f0ec-28e2364b1f5b@probesys.com> References: <9cc1bd8f-ed6c-60e8-f0ec-28e2364b1f5b@probesys.com> Message-ID: <3b0f71ab-aa02-46a8-8b5b-44e21c4e85fd@www.fastmail.com> On Tue, Jan 28, 2020, at 10:22 AM, Zorg wrote: > put every time i this to add event (thunderbird, evolution, curl ) i > have this in the log "HTTP/1.1 500 Internal Server Error" (error=The > server encountered an internal error.) Do you see any indicative error message in the log file (e.g. syslog)? Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Tue Jan 28 04:37:42 2020 From: rsto at fastmailteam.com (Robert Stepanek) Date: Tue, 28 Jan 2020 10:37:42 +0100 Subject: Database upgrade and Xapian version dependency In-Reply-To: References: Message-ID: On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: > Just for having it slightly clearer? When you upgrade the Cyrus version and the version you are upgrading to is a too close one? for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both? is it needed to launch (or recommended for some reason) the final upgrade commands : > > reconstruct -V max > ctl_conversationsdb -b -r > quota -f I'm not sure. Perhaps @ellie could answer that? > By the way, does exist any kind of Xapian needed version for the last 3.0.13 version?. I?m running Xapian 1.4.9, it?s pretty new? One feature that's missing in Xapian 1.4 is improved support for Chinese and Japanese snippet generation. If you don't need that, you should be fine with 1.4. Otherwise I suggest to use either Xapian upstream master, or our cyruslibs copy at https://github.com/cyrusimap/cyruslibs Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From zorg at probesys.com Tue Jan 28 05:10:41 2020 From: zorg at probesys.com (Zorg) Date: Tue, 28 Jan 2020 11:10:41 +0100 Subject: Create ics in another calendar (internal server error) In-Reply-To: <3b0f71ab-aa02-46a8-8b5b-44e21c4e85fd@www.fastmail.com> References: <9cc1bd8f-ed6c-60e8-f0ec-28e2364b1f5b@probesys.com> <3b0f71ab-aa02-46a8-8b5b-44e21c4e85fd@www.fastmail.com> Message-ID: <72ec0109-ae4c-087f-74b6-3249566d4861@probesys.com> Hello, Nothing more in my log I manage to solve this issue in the debian changelog i have read that cyrus-imapd (3.0.8-6+deb10u3) buster-security; urgency=medium ? * Add patch to avoid mailbox creation as administrator ??? (Closes: #CVE-2019-19783) So I have try to create the mail and the default calendar folder for myadmin Now it work Can't really explain without reading the code but it work (after some tests it work with only mailbox) So what i understand is that to write events in another calendar Thank Le 28/01/2020 ? 10:26, Robert Stepanek a ?crit?: > On Tue, Jan 28, 2020, at 10:22 AM, Zorg wrote: >> put every time i this to add event (thunderbird, evolution, curl ) i >> have this in the log? "HTTP/1.1 500 Internal Server Error" (error=The >> server encountered an internal error.) > > Do you see any indicative error message in the log file (e.g. syslog)? > > Cheers, > Robert > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at ramattack.net Tue Jan 28 14:04:59 2020 From: egoitz at ramattack.net (Egoitz Aurrekoetxea) Date: Tue, 28 Jan 2020 20:04:59 +0100 Subject: Database upgrade and Xapian version dependency In-Reply-To: References: Message-ID: <3C2A1BAC-A63D-4C50-BF5F-61F6AEAEC4A9@ramattack.net> Hi Robert :) Thanks a lot for your answer mate :) :) Very thankful :) :) I think for the moment we?ll stay in the 1.4?. this is not going to be a big problem? And about versions? it would be fine if Ellie could told? but I assume there?s no problem with that due to not having database upgrades of mailboxes at least? so I assume quota and reconstruct won?t be needed? the conversations? I?ll take a look at them? to see if some important changes have happen? but I suppose there have not really existed? Bye !! :) > El 28 ene 2020, a las 10:37, Robert Stepanek escribi?: > > On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: >> Just for having it slightly clearer? When you upgrade the Cyrus version and the version you are upgrading to is a too close one? for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both? is it needed to launch (or recommended for some reason) the final upgrade commands : >> >> reconstruct -V max >> ctl_conversationsdb -b -r >> quota -f > > I'm not sure. Perhaps @ellie could answer that? > >> By the way, does exist any kind of Xapian needed version for the last 3.0.13 version?. I?m running Xapian 1.4.9, it?s pretty new? > > One feature that's missing in Xapian 1.4 is improved support for Chinese and Japanese snippet generation. If you don't need that, you should be fine with 1.4. Otherwise I suggest to use either Xapian upstream master, or our cyruslibs copy at https://github.com/cyrusimap/cyruslibs > > Cheers, > Robert > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Tue Jan 28 18:08:39 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 29 Jan 2020 10:08:39 +1100 Subject: Database upgrade and Xapian version dependency In-Reply-To: <3C2A1BAC-A63D-4C50-BF5F-61F6AEAEC4A9@ramattack.net> References: <3C2A1BAC-A63D-4C50-BF5F-61F6AEAEC4A9@ramattack.net> Message-ID: >> On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: >>> Just for having it slightly clearer? When you upgrade the Cyrus version and the version you are upgrading to is a too close one? for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both? is it needed to launch (or recommended for some reason) the final upgrade commands : >>> >>> reconstruct -V max >>> ctl_conversationsdb -b -r >>> quota -f As long as you're looking at 3.x.x and higher*, then if it's a stable release (where only the third number has changed, e.g. 3.0.8->3.0.13) you shouldn't need to. We don't do big world-breaking/data-format-changing changes in stable releases like this. That said, we might have fixed a bug between the versions, and your existing data might be bad due to the bug that was fixed, and you might need to run commands like these after upgrading to the fixed version to repair the bad data. If I know this is the case, I'll say so in the release notes (so check those, including for the intermediate releases if you're skipping over some). But I might not know until someone upgrades and reports that they needed to do it. So, if it doesn't say you need to do it, you probably don't need to do it. But if you don't do it, and things like quota or conversations seem weird after the upgrade, try these sort of commands. And if it fixes it, let us know (on the mailing list or a github issue) so we can retcon the release notes to mention it for the next person. :) * For people looking at 2.5 and earlier, this pattern doesn't necessarily hold, and I don't know enough about those older versions to provide general advice. If in doubt, ask the mailing list, and if you can run a recent version (that's closer to what's in Fastmail's organisational memory), that's your best bet ;) Cheers, ellie -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Jan 28 20:35:26 2020 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Wed, 29 Jan 2020 02:35:26 +0100 Subject: Database upgrade and Xapian version dependency In-Reply-To: References: <3C2A1BAC-A63D-4C50-BF5F-61F6AEAEC4A9@ramattack.net> Message-ID: <7EAD3749-BB5B-44DB-88D3-5476E83DAD3C@sarenet.es> Hi!! Thanks a lot for your nice answer :) . Totally logical, reasonable and clear answer. Nicely explained. I?ll share in the list all the relevant situations I?ll find. Thanks again ;) Egoitz, > El 29 ene 2020, a las 0:08, ellie timoney escribi?: > >>>> On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: >>>> Just for having it slightly clearer? When you upgrade the Cyrus version and the version you are upgrading to is a too close one? for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both? is it needed to launch (or recommended for some reason) the final upgrade commands : >>>> >>>> reconstruct -V max >>>> ctl_conversationsdb -b -r >>>> quota -f > > As long as you're looking at 3.x.x and higher*, then if it's a stable release (where only the third number has changed, e.g. 3.0.8->3.0.13) you shouldn't need to. We don't do big world-breaking/data-format-changing changes in stable releases like this. > > That said, we might have fixed a bug between the versions, and your existing data might be bad due to the bug that was fixed, and you might need to run commands like these after upgrading to the fixed version to repair the bad data. If I know this is the case, I'll say so in the release notes (so check those, including for the intermediate releases if you're skipping over some). But I might not know until someone upgrades and reports that they needed to do it. > > So, if it doesn't say you need to do it, you probably don't need to do it. But if you don't do it, and things like quota or conversations seem weird after the upgrade, try these sort of commands. And if it fixes it, let us know (on the mailing list or a github issue) so we can retcon the release notes to mention it for the next person. :) > > * For people looking at 2.5 and earlier, this pattern doesn't necessarily hold, and I don't know enough about those older versions to provide general advice. If in doubt, ask the mailing list, and if you can run a recent version (that's closer to what's in Fastmail's organisational memory), that's your best bet ;) > > Cheers, > > ellie > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From karagian at gmail.com Wed Jan 29 05:38:43 2020 From: karagian at gmail.com (Savvas Karagiannidis) Date: Wed, 29 Jan 2020 12:38:43 +0200 Subject: Cross script vulnerability (XSS) in httpd Message-ID: Hi, not sure if this is an actual issue, so I'm posting it here first, in case someone knows better. We recently ran a vulnerability assessment using nessus against our server running cyrus and it detected the following medium risk XSS issue (the actual report is at the bottom of the email) 9080 is the custom port https is configured to listen on. >From what I understand it seems that someone could craft a special request and enter script code via the headers sent, code that appears in the response and could actually be executed in case a browser is used. The report had multiple example requests, but technically they were all the same, so I'm just attaching the first example request that confirms the issue. Regards, Savvas Karagiannidis Here's the related part of the report: Synopsis The remote web server is affected by a cross-site scripting vulnerability. Description The remote host is running a web server that fails to adequately sanitize request strings of malicious JavaScript. A remote attacker can exploit this issue, via a specially crafted request, to execute arbitrary HTML and script code in a user's browser within the security context of the affected site. See Also https://en.wikipedia.org/wiki/Cross-site_scripting Solution Contact the vendor for a patch or upgrade. Risk Factor Medium CVSS Base Score 4.3 (CVSS2#AV:N/AC:M/Au:N/C:N/I:P/A:N) CVSS Temporal Score 3.7 (CVSS2#E:H/RL:OF/RC:C) References BID 5011 BID 5305 BID 7344 BID 7353 BID 8037 BID 14473 BID 17408 BID 54344 CVE CVE-2002-1060 CVE CVE-2002-1700 CVE CVE-2003-1543 CVE CVE-2005-2453 CVE CVE-2006-1681 CVE CVE-2012-3382 XREF CWE:79 Plugin Information Published: 2001/11/30, Modified: 2018/07/06 Plugin Output tcp/9080 ------------------------------ Request #1 ------------------------------ The full request used to detect this flaw was : GET /cgi-bin/llknxx7s.html HTTP/1.1 Host: :9080 Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1 Accept-Language: en Connection: Close User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0) Pragma: no-cache Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */* The output was : HTTP/1.1 404 Not Found Date: Thu, 23 Jan 2020 18:13:22 GMT Connection: close, Upgrade Upgrade: Vary: Accept-Encoding Content-Type: text/html; charset=utf-8 Content-Length: 437 [...] Jansson/2.9 Server at Port 9080