From chose at ajetaci.cz Wed Jan 2 03:20:59 2019 From: chose at ajetaci.cz (chose) Date: Wed, 02 Jan 2019 09:20:59 +0100 Subject: Cyrus 2.4 and unexpunge messages. Message-ID: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> Good morning, I've unexpunged messages in the mail box, all is recovered but the flag "deleted" persist, so Roundcube see the email as deleted and the emails are grey. Did I missed some step to full recover emails ? Thanks and best regards J.Karliak -- Ma domena pouziva zabezpeceni a kontrolu SPF (www.openspf.org) a DomainKeys/DKIM (s ADSP) a implementaci DMARC. Pokud mate problemy s dorucenim emailu, zacnete pouzivat metody overeni puvody emailu zminene vyse. Dekuji. My domain use SPF (www.openspf.org) and DomainKeys/DKIM (with ADSP) policy and implementation of the DMARC. If you've problem with sending emails to me, start using email origin methods mentioned above. Thank you. From chose at ajetaci.cz Wed Jan 2 03:35:04 2019 From: chose at ajetaci.cz (chose) Date: Wed, 02 Jan 2019 09:35:04 +0100 Subject: Cyrus 2.4 and unexpunge messages. In-Reply-To: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> References: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> Message-ID: <4a16ef849ffa28984ee7b54615303776@ajetaci.cz> Ohh, I found it - I had to delete cyrus.* files in the recovered mailbox, after it I see all emails unreaded. Better than "deleted" :). Or is there another/better way ? Thanks and best regards J.Karliak Dne 2019-01-02 09:20, chose napsal: > Good morning, > I've unexpunged messages in the mail box, all is recovered but the > flag "deleted" persist, so Roundcube see the email as deleted and the > emails are grey. > Did I missed some step to full recover emails ? > Thanks and best regards > J.Karliak -- Ma domena pouziva zabezpeceni a kontrolu SPF (www.openspf.org) a DomainKeys/DKIM (s ADSP) a implementaci DMARC. Pokud mate problemy s dorucenim emailu, zacnete pouzivat metody overeni puvody emailu zminene vyse. Dekuji. My domain use SPF (www.openspf.org) and DomainKeys/DKIM (with ADSP) policy and implementation of the DMARC. If you've problem with sending emails to me, start using email origin methods mentioned above. Thank you. From dpc22 at cam.ac.uk Wed Jan 2 04:00:02 2019 From: dpc22 at cam.ac.uk (David Carter) Date: Wed, 02 Jan 2019 09:00:02 +0000 Subject: Cyrus 2.4 and unexpunge messages. In-Reply-To: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> References: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> Message-ID: <1e8cb91276a65e6902673358dfac6c50@cam.ac.uk> On 2019-01-02 08:20, chose wrote: > Good morning, > I've unexpunged messages in the mail box, all is recovered but the > flag "deleted" persist, so Roundcube see the email as deleted and the > emails are grey. > Did I missed some step to full recover emails ? The unexpunge command has a "-d" option: -d Unset the \Deleted flag on any restored messages. -- David Carter Email: dpc22 at cam.ac.uk University of Cambridge, Phone: (01223) 748408 Information Services, Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB From awilliam at whitemice.org Wed Jan 2 08:31:56 2019 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 02 Jan 2019 08:31:56 -0500 Subject: Cyrus 2.4 and unexpunge messages. In-Reply-To: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> References: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> Message-ID: <1546435916.3213.1.camel@whitemice.org> On Wed, 2019-01-02 at 09:20 +0100, chose wrote: > I've unexpunged messages in the mail box, all is recovered but the? > flag "deleted" persist, so Roundcube see the email as deleted and > the?emails are grey. Yes, this is correct. Unexpunge unexpunges, it does not undelete [delete in IMAP being a flag]. This a feature, not a bug [IMAP handles deletes in a consistent, reliable, sane, standard way vs. the hackish behavior implemented by most MUAs]. > ???Did I missed some step to full??recover emails ? They are fully recovered; you can mark them as undeleted via the client. -- Executive Committee Vice-Chair Michigan Association of Railroad Passengers 537 Shirley St NE Grand Rapids, MI 49503-1754 Phone: 616.581.8010 E-mail: awilliam at whitemice.org GPG#D95ED383 From boutilpj at ednet.ns.ca Wed Jan 2 10:05:29 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 2 Jan 2019 11:05:29 -0400 Subject: Cyrus 2.4 and unexpunge messages. In-Reply-To: <1546435916.3213.1.camel@whitemice.org> References: <902ac9a65199c14ea55cd5a45e5513c9@ajetaci.cz> <1546435916.3213.1.camel@whitemice.org> Message-ID: On 1/2/19 9:31 AM, Adam Tauno Williams wrote: > On Wed, 2019-01-02 at 09:20 +0100, chose wrote: >> I've unexpunged messages in the mail box, all is recovered but the >> flag "deleted" persist, so Roundcube see the email as deleted and >> the?emails are grey. > > Yes, this is correct. Unexpunge unexpunges, it does not undelete > [delete in IMAP being a flag]. This a feature, not a bug [IMAP handles > deletes in a consistent, reliable, sane, standard way vs. the hackish > behavior implemented by most MUAs]. > >> ???Did I missed some step to full??recover emails ? Run unexpunge with -d -d Unset the \Deleted flag on any restored messages. > > They are fully recovered; you can mark them as undeleted via the > client. > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From egoitz at sarenet.es Thu Jan 3 07:27:26 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 13:27:26 +0100 Subject: www.cyrusimap.org down? Message-ID: Good afternoon, I'm suffering connectivity issues to Cyrus imap web site. Perhaps some kind of problem at hosting side?. It takes from yesterday... do we know something about when will it be up again?. Cheers! -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 08:53:30 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 14:53:30 +0100 Subject: Question for upgrading In-Reply-To: <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> Message-ID: <4d260681b7e89b934237a64ade5d4502@sarenet.es> Hi Nic, Sorry for the delay answering... I have been ill and later on holiday, with very very very few time to answer to nothing when coming from illness. Sorry then... My idea really, was to upgrade to 3.0... So I'm planning: - Upgrade in place from 2.3 to 2.4. - Continue giving the service... - Later setup a 3.0 slave.... all of this WITHOUT doing a reconstruct -V max..... as it seems in 2.4 all works (unexpunge command and so) without it... should say too, it seems the own sync client performs database upgrade.... Jan 3 14:46:02 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Alberto (10 -> 12) Jan 3 14:46:06 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Comerciales (10 -> 12) Jan 3 14:46:15 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Instalaciones-staff (10 -> 12) Jan 3 14:46:25 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.RRHH (10 -> 12) It seems to anyway be happening.... - When the 3.0 slave is totally replicated from the 2.4 mater... turn the 3.0 as mater, and take a copy of it becoming it slave..... This way I should have all the platform upgraded without downtimes.... How do you see Nic?. I think it should work... I'm still doing checks anyway..... Cheers!! Thanks a lot for your time!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [2] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 15-12-2018 17:05, Nic Bernstein escribi?: > On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: > >> I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem is that this reconstruct command, takes ages and I'm not able to keep the service offline so many time. So I have been thinking in the following scenario : > > Egoitz, > As long as you've followed all of the various steps needed to account for version changes, as outlined in the Release Notes for _all_ intermediary releases, then the last step should be the updating of the indexes. Rather than running "reconstruct -V max" on all mailboxes at once, simply run it on subsets. We've done this on large installations without ill effect. You can have the service on line during the reconstruct, and, as you note, have most all functionality present. > > We build a list of users (mboxlist.txt), and then run a cron job, during off-peak hours, using the 'parallel' command like so (from 'crontab -e -u cyrus'): > >> ### >> ## Start a batch of recursive reconstruct jobs at 7PM and discontinue >> ## at 6:53AM. >> 0 19 * * * parallel -j 5 --resume --joblog /var/lib/imap/reconstruct.log cyrus reconstruct -G -V max -r -u {} < /var/lib/imap/mboxlist.txt >> 53 06 * * * kill -TERM `ps ax | grep [p]arallel | awk '{print $1}'` > Cheers, > -nic > > -- > Nic Bernstein nic at onlight.com > Onlight Inc. www.onlight.com [1] > 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 > Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 Links: ------ [1] http://www.onlight.com [2] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 08:54:50 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 14:54:50 +0100 Subject: Question for upgrading In-Reply-To: <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> Message-ID: <3ee0528d34e634f32be77a69d5ea7df5@sarenet.es> Thank you so much Binarus!!! And as said before, sorry for the big delay!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 18-12-2018 12:15, Binarus escribi?: > On 15.12.2018 17:05, Nic Bernstein wrote: On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: > I was trying to upgrade part of our Cyrus imap installation, > concretely that one consisting in still 2.3. I was planning to set up > Cyrus 3.0. I have seen all works properly except for the unexpunge > command because as someone stated here, a reconstruct -V max was > needed.The problem is that this reconstruct command, takes ages and > I'm not able to keep the service offline so many time. So I have been > thinking in the following scenario : > > Egoitz, > As long as you've followed all of the various steps needed to account > for version changes, as outlined in the Release Notes for /all/ > intermediary releases, then the last step should be the updating of the > indexes. Rather than running "reconstruct -V max" on all mailboxes at > once, simply run it on subsets. We've done this on large installations > without ill effect. You can have the service on line during the > reconstruct, and, as you note, have most all functionality present. In my case, before the reconstruct had finished, I had several problems which might be not acceptable for large organizations. For example, users could not move messages between folders in their mailbox. I would consider this quite basic functionality, because deleting a message (with most clients) also means moving it (to trash). Functionality was back not before the reconstruct had finished completely. If interested, please see the respective thread from yesterday / today for details (I don't want to clutter the list by repeating all that stuff here). Regards, Binarus ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 08:56:26 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 14:56:26 +0100 Subject: Question for upgrading In-Reply-To: <292a2603-3ea3-6a5f-f5a0-0028cd9f36a8@jangulo.net> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> <292a2603-3ea3-6a5f-f5a0-0028cd9f36a8@jangulo.net> Message-ID: Hi Javier!, Sorry for the delay and many many thanks for your ideas :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 18-12-2018 13:50, Javier Angulo escribi?: > On 12/18/18 12:15 PM, Binarus wrote: On 15.12.2018 17:05, Nic Bernstein wrote: On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: I was trying to upgrade part of our Cyrus imap installation, > concretely that one consisting in still 2.3. I was planning to set up > Cyrus 3.0. I have seen all works properly except for the unexpunge > command because as someone stated here, a reconstruct -V max was > needed.The problem is that this reconstruct command, takes ages and > I'm not able to keep the service offline so many time. So I have been > thinking in the following scenario : > Egoitz, > As long as you've followed all of the various steps needed to account > for version changes, as outlined in the Release Notes for /all/ > intermediary releases, then the last step should be the updating of the > indexes. Rather than running "reconstruct -V max" on all mailboxes at > once, simply run it on subsets. We've done this on large installations > without ill effect. You can have the service on line during the > reconstruct, and, as you note, have most all functionality present. In my case, before the reconstruct had finished, I had several problems which might be not acceptable for large organizations. For example, users could not move messages between folders in their mailbox. I would consider this quite basic functionality, because deleting a message (with most clients) also means moving it (to trash). Functionality was back not before the reconstruct had finished completely. Apart from those we also had some weird problems with the message sort order (using roundcube) before reconstruct was run. We split the mailboxes reconstruction into 8 parallel jobs (IO on the mailbox spool is the limit if you go from 2.3 -> 2.4/3.0). If you ran the reconstruction online, once finished I would suggest to check again all indexes version (some reconstruction jobs fail). Cheers ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake at ispn.net Thu Jan 3 09:00:36 2019 From: blake at ispn.net (Blake Hudson) Date: Thu, 3 Jan 2019 08:00:36 -0600 Subject: www.cyrusimap.org down? In-Reply-To: References: Message-ID: <8747275f-4c68-2208-4f57-7b4b47ea4f5f@ispn.net> Ditto. Noticed the issue yesterday afternoon. C:\Users\blake>ping www.cyrusimap.org Pinging WWW-01.cyrusimap.org [128.2.12.195] with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 128.2.12.195: ??? Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), C:\Users\blake>telnet www.cyrusimap.org 80 Connecting To www.cyrusimap.org...Could not open connection to the host, on port 80: Connect failed C:\Users\blake>tracert www.cyrusimap.org Tracing route to WWW-01.cyrusimap.org [128.2.12.195] over a maximum of 30 hops: ? 4??? <1 ms??? <1 ms??? <1 ms 192-155-77-33.static.kc.surewest.net [192.155.77.33] ? 5???? 1 ms??? <1 ms??? <1 ms? 126-0-212.ksle.everestkc.net [64.126.0.212] ? 6??? <1 ms??? <1 ms??? <1 ms 64-126-2-106.static.everestkc.net [64.126.2.106] ? 7???? 2 ms???? 1 ms???? 1 ms be4163.ccr21.mci01.atlas.cogentco.com [38.104.86.101] ? 8??? 14 ms??? 15 ms??? 13 ms be2831.ccr41.ord01.atlas.cogentco.com [154.54.42.166] ? 9??? 21 ms??? 22 ms??? 22 ms be2717.ccr21.cle04.atlas.cogentco.com [154.54.6.222] ?10??? 25 ms??? 24 ms??? 24 ms be2821.rcr21.pit02.atlas.cogentco.com [154.54.83.118] ?11??? 25 ms??? 25 ms??? 25 ms te0-0-2-3.nr11.b015486-1.pit02.atlas.cogentco.com [154.24.50.198] ?12??? 32 ms??? 32 ms??? 32 ms? 38.140.44.154 ?13??? 32 ms??? 32 ms??? 32 ms? CORE255-POD-I-CYH.GW.CMU.NET [128.2.255.249] ?14??? 32 ms??? 32 ms??? 32 ms? POD-D-CYH-CORE255.GW.CMU.NET [128.2.255.202] ?15???? *??????? *??????? *???? Request timed out. ?16???? *??????? *??????? *???? Request timed out. ?17???? *??????? *??????? *???? Request timed out. ?18???? *??????? *??????? *???? Request timed out. ?19???? *??????? *??????? *???? Request timed out. ?20???? *??????? *??????? *???? Request timed out. Egoitz Aurrekoetxea wrote on 1/3/2019 6:27 AM: > > Good afternoon, > > > I'm suffering connectivity issues to Cyrus imap web site. Perhaps some > kind of problem at hosting side?. It takes from yesterday... do we > know something about when will it be up again?. > > > Cheers! > > -- > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 09:55:28 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 15:55:28 +0100 Subject: Xapian searches in Cyrus 3.0 Message-ID: Good afternoon, I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Thu Jan 3 10:08:44 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Thu, 03 Jan 2019 16:08:44 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: References: Message-ID: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> Hello, On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: > I was planning to perform mailboxes squattering in rolling mode. Have > you had some kind of expience on searching with Xapian and IMAP > protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it > with IMAP is not so advantageous?. Both IMAP and JMAP search share the same core search API in Cyrus, so all search features available in JMAP can also be used using IMAP. The key is to use FUZZY search for text search, and this can be an advantage especially for non-English search over the other search backends. The complete list of attributes to search is in this C file: https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/search_expr.c#L2235 Any attribute that is marked SEA_FUZZABLE will get stored in Xapian and is available for stem-based search. I'm not sure if there's better documentation on the non-standard fields, but if you have any questions feel free to ask! Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 10:27:20 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 16:27:20 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> References: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> Message-ID: Thank you so much Robert!!!! Many many thanks!!! Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 03-01-2019 16:08, Robert Stepanek escribi?: > Hello, > > On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: > >> I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. > > Both IMAP and JMAP search share the same core search API in Cyrus, so all search features available in JMAP can also be used using IMAP. The key is to use FUZZY search for text search, and this can be an advantage especially for non-English search over the other search backends. > > The complete list of attributes to search is in this C file: https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/search_expr.c#L2235 Any attribute that is marked SEA_FUZZABLE will get stored in Xapian and is available for stem-based search. > > I'm not sure if there's better documentation on the non-standard fields, but if you have any questions feel free to ask! > > Cheers, > Robert > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 10:32:16 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 16:32:16 +0100 Subject: Question about manual replication (-u ) Message-ID: <9797216371bb00cdb59351370d5607d2@sarenet.es> Good afternoon, Is it possible to launch several instances of "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. Doing it just one mailbox at a time takes ages.... It would help me a lot, the fact of parallelizing and have no disk bottleneck issues.... I think it should be possible... isn't it?. Perhaps it just allowed between same version in source and dest?. Or can be done for instance too, with a 2.4 as master to 3.0 slave?. Cheers!! -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Thu Jan 3 10:37:18 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 03 Jan 2019 16:37:18 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> References: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> Message-ID: <7A6601EA26F2199AEF54140C@tyrion.rrz.uni-koeln.de> --On 3. Januar 2019 um 16:08:44 +0100 Robert Stepanek wrote: > On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: >> I was planning to perform mailboxes squattering in rolling mode. Have >> you had some kind of expience on searching with Xapian and IMAP >> protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it >> with IMAP is not so advantageous?. > Both IMAP and JMAP search share the same core search API in Cyrus, so > all search features available in JMAP can also be used using IMAP. The > key is to use FUZZY search for text search, and this can be an advantage > especially for non-English search over the other search backends. A few additional comments: With the setting "search_fuzzy_always: 1" all searches use Xapian. Otherwise searches from clients that don't use FUZZY run completely without a search index. There is no fallback mechanism to squatter. You also need to be careful with the conversationsdb. In my experience the DB is not updated when syncing from a 2.4 server to a 3.0 server. That means that you need ro rebuild each user's conversationdb manually using ctl_conversationsdb. -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From egoitz at sarenet.es Thu Jan 3 11:06:19 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 17:06:19 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <7A6601EA26F2199AEF54140C@tyrion.rrz.uni-koeln.de> References: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> <7A6601EA26F2199AEF54140C@tyrion.rrz.uni-koeln.de> Message-ID: <75e0c2d1c9504d128f00a94bdfd654fa@sarenet.es> Hi Sebastian! I suppose I won't have problems when upgrading, because I'll do a 2.4 in place upgrade (with no xapian, no indexing) and later a replication to a 3.0 Cyrus with Xapian enabled, where all indexes due to this sync should being generated (it seems logs say that at least...). Is Xapian search slower than "previous way" in some situation?. When talking about a fallback, I meant you know, running Squatter without being in rolling mode and behaving and indexing the way before Xapian appeared... which I suppose it would be used berkeley or similar, but have seen no berkeley support is present in last Cyrus versions so... don't really know which database backend where used... I suppose to some other database type.... Thanks mate!!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 03-01-2019 16:37, Sebastian Hagedorn escribi?: > --On 3. Januar 2019 um 16:08:44 +0100 Robert Stepanek wrote: > > On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: I was planning to perform mailboxes squattering in rolling mode. Have > you had some kind of expience on searching with Xapian and IMAP > protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it > with IMAP is not so advantageous?. Both IMAP and JMAP search share the same core search API in Cyrus, so > all search features available in JMAP can also be used using IMAP. The > key is to use FUZZY search for text search, and this can be an advantage > especially for non-English search over the other search backends. A few additional comments: With the setting "search_fuzzy_always: 1" all searches use Xapian. Otherwise searches from clients that don't use FUZZY run completely without a search index. There is no fallback mechanism to squatter. You also need to be careful with the conversationsdb. In my experience the DB is not updated when syncing from a 2.4 server to a 3.0 server. That means that you need ro rebuild each user's conversationdb manually using ctl_conversationsdb. ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Thu Jan 3 14:05:16 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 03 Jan 2019 20:05:16 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <75e0c2d1c9504d128f00a94bdfd654fa@sarenet.es> References: <1546528124.815968.1624537528.151634A6@webmail.messagingengine.com> <7A6601EA26F2199AEF54140C@tyrion.rrz.uni-koeln.de> <75e0c2d1c9504d128f00a94bdfd654fa@sarenet.es> Message-ID: <2D09516BCF753E870B59C465@Sebbis-iMac.local> Sorry, but no. You are right that the *indexes* are generated properly when you sync, but I was talking about *conversationsdb*. That is *not* updated when syncing from 2.4 to 3.0. You need ro run ctl_conversationsdb for each user, otherwise Xapian won't find any messages. ctl_conversationsdb -z USER ctl_conversationsdb -b USER > I suppose I won't have problems when upgrading, because I'll do a 2.4 in > place upgrade (with no xapian, no indexing) and later a replication to a > 3.0 Cyrus with Xapian enabled, where all indexes due to this sync should > being generated (it seems logs say that at least...). Is Xapian search > slower than "previous way" in some situation?. When talking about a > fallback, I meant you know, running Squatter without being in rolling > mode and behaving and indexing the way before Xapian appeared... which I > suppose it would be used berkeley or similar, but have seen no berkeley > support is present in last Cyrus versions so... don't really know which > database backend where used... I suppose to some other database type.... > > > Thanks mate!!! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario > hacerlo. > > El 03-01-2019 16:37, Sebastian Hagedorn escribi?: > >> --On 3. Januar 2019 um 16:08:44 +0100 Robert Stepanek >> wrote: >> >> On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: I was >> planning to perform mailboxes squattering in rolling mode. Have you had >> some kind of expience on searching with Xapian and IMAP protocol?. >> Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is >> not so advantageous?. Both IMAP and JMAP search share the same core >> search API in Cyrus, so all search features available in JMAP can also >> be used using IMAP. The key is to use FUZZY search for text search, and >> this can be an advantage especially for non-English search over the >> other search backends. > > A few additional comments: > > With the setting "search_fuzzy_always: 1" all searches use Xapian. > Otherwise searches from clients that don't use FUZZY run completely > without a search index. There is no fallback mechanism to squatter. You > also need to be careful with the conversationsdb. In my experience the > DB is not updated when syncing from a 2.4 server to a 3.0 server. That > means that you need ro rebuild each user's conversationdb manually using -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 262 bytes Desc: not available URL: From egoitz at sarenet.es Mon Jan 7 03:59:35 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 09:59:35 +0100 Subject: Xapian searches of the body of an email Message-ID: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Good morning, I have been testing Xapian searches. Have seen, it's not able to find strings inside the body of the email. If I set in imap.conf "search_fuzzy_always: 1" no content is displayed in the searches of a Roundcube stock webmail. If I remove that config value from imap.conf and restart services, then search results appear. Does Xapian not index the body of emails?. Does Xapian, just index the headers?. But this affirmation does not seem to be possible in my case too... as I have in the config "search_index_headers: no". I'm using the following config : conversations: 1 search_engine: xapian search_index_headers: no search_batchsize: 8192 defaultsearchtier: t1 t1searchpartition-default: /expert/search t1searchpartition-expert2: /expert2/search t1searchpartition-expert3: /expert3/search Could anyone help me mates?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Mon Jan 7 04:19:08 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 10:19:08 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: That sounds like the conversationsdb issue I was talking about. Have you tried these steps? ctl_conversationsdb -z USER ctl_conversationsdb -b USER > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 262 bytes Desc: not available URL: From egoitz at sarenet.es Mon Jan 7 08:51:02 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 14:51:02 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Hi mate! This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... Thanks a lot!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > >> I have been testing Xapian searches. Have seen, it's not able to find >> strings inside the body of the email. If I set in imap.conf >> "search_fuzzy_always: 1" no content is displayed in the searches of a >> Roundcube stock webmail. If I remove that config value from imap.conf >> and restart services, then search results appear. Does Xapian not index >> the body of emails?. Does Xapian, just index the headers?. But this >> affirmation does not seem to be possible in my case too... as I have in >> the config "search_index_headers: no". >> >> I'm using the following config : >> >> conversations: 1 >> search_engine: xapian >> search_index_headers: no >> search_batchsize: 8192 >> defaultsearchtier: t1 >> t1searchpartition-default: /expert/search >> t1searchpartition-expert2: /expert2/search >> t1searchpartition-expert3: /expert3/search >> >> Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 09:05:52 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 15:05:52 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> And, by the way.... when using Squatter instead of Xapian as a search engine.... what do we really lost?. Just the fact of having a statistical worse results?. Is it Xapian faster than squat engine?. Sorry for having so many questions but... I suppose I don't have the implications of each one totally clear :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 14:51, Egoitz Aurrekoetxea escribi?: > Hi mate! > > This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... > > Thanks a lot!! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 09:41:59 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 15:41:59 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> Message-ID: <776585952e420b671ed40ed7495e88db@sarenet.es> Sorry for asking again.. ctl_conversationsdb -z and -b (with -r I assume for all users) should be run only on new user accounts or... periodically for any user account?. Or does squatter maintain too the conversations database?. Best regards, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 15:05, Egoitz Aurrekoetxea escribi?: > And, by the way.... > > when using Squatter instead of Xapian as a search engine.... what do we really lost?. Just the fact of having a statistical worse results?. Is it Xapian faster than squat engine?. > > Sorry for having so many questions but... I suppose I don't have the implications of each one totally clear :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 14:51, Egoitz Aurrekoetxea escribi?: > > Hi mate! > > This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... > > Thanks a lot!! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Mon Jan 7 09:57:39 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 15:57:39 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Hi, --On 7. Januar 2019 um 14:51:02 +0100 Egoitz Aurrekoetxea wrote: > This seems to take ages... why don't you run it for a single account first, to make sure that it actually helps? >I'm trying to figure the best way of > implementing this and of clarifying concepts.... I'm running Squatter in > rolling replication mode and exist the concept of conversations then. > What is the exact role of each of them?. Squatter seems to index the > mailbox but when something is not properly indexed instead of running > Squatter you use ctl_conversations for reindexing some part again or... squatter is nowadays a bit of a misnomer, because it uses whatever index you have configured. In cyrus 2.4, squatter would always create a SQUAT index. When you run squatter with Xapian, it will build the index, but for the index to actually work, you also need the conversationsdb. squatter does not touch the conversationsdb! The index is only a pointer to the conversationsdb, not to actual messages. Robert can probably explain this much better than me, but I think the problem is the following: ? when you have conversations enabled in imapd.conf, normal deliveries to the mailboxes (e.g. using lmtpd) will update the conversationsdb ? syncing (at least using the "old" mechanism, not sure about sync between instances running 3.0) does *not* update the conversationsdb Once you have a running 3.0 server, you probably won't need to run ctl_conversationsdb ever again. But when you are at the stage of syncing mail from 2.4 to 3.0, you *will* need to rebuild each user's conversationdb at least once, after you have finished with syncing that user. Again, this is all based on my understanding and not an official answer. > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > >> That sounds like the conversationsdb issue I was talking about. Have you >> tried these steps? >> >> ctl_conversationsdb -z USER >> ctl_conversationsdb -b USER Mit freundlichen Gr??en Sebastian Hagedorn -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From Hagedorn at uni-koeln.de Mon Jan 7 10:01:20 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 16:01:20 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> Message-ID: <9EC01839DEEE8F7A28BAFF0D@tyrion.rrz.uni-koeln.de> That's a really good question. I had been hoping that the performance of a Xapian search would be much better than a SQUAT search, but now I'm not so sure. --On 7. Januar 2019 um 15:05:52 +0100 Egoitz Aurrekoetxea wrote: > And, by the way.... > > when using Squatter instead of Xapian as a search engine.... what do we > really lost?. Just the fact of having a statistical worse results?. Is > it Xapian faster than squat engine?. -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From Hagedorn at uni-koeln.de Mon Jan 7 10:03:16 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 16:03:16 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <776585952e420b671ed40ed7495e88db@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> <776585952e420b671ed40ed7495e88db@sarenet.es> Message-ID: <94BFD098ADE0B30BCCC8CBFF@tyrion.rrz.uni-koeln.de> As I wrote before, this should only be necessary as long as you are in the syncing stage of the migration. Once all new mail is delivered to the 3.0 server everything should just work ? I think ;-) --On 7. Januar 2019 um 15:41:59 +0100 Egoitz Aurrekoetxea wrote: > ctl_conversationsdb -z and -b (with -r I assume for all users) should be > run only on new user accounts or... periodically for any user account?. > Or does squatter maintain too the conversations database?. -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From elias at hi.is Mon Jan 7 10:13:54 2019 From: elias at hi.is (=?utf-8?B?RWzDrWFzIEhhbGxkw7NyIMOBZ8O6c3Rzc29u?=) Date: Mon, 7 Jan 2019 15:13:54 +0000 Subject: Xapian searches of the body of an email In-Reply-To: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Regarding indexing and searching in body of emails; what if the body text is encoded in base64 or quoted-printable? It won't yield any unencoded search strings, or what? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Mon Jan 7 10:42:48 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Mon, 07 Jan 2019 16:42:48 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> Hi, Sebastian is right: On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: > > squatter is nowadays a bit of a misnomer, because it uses > whatever index> you have configured. In cyrus 2.4, squatter would always create a > SQUAT> index. When you run squatter with Xapian, it will build the > index, but for> the index to actually work, you also need the conversationsdb. conversations.db is indeed a misnomer now. The database was only used to keep track of mail threads (hence the name), but its role expanded. One of the indexes it stores is the SHA1 hashes of every message, and separate hashes for each of that message MIME parts. Such a hash is named the GUID, and for each GUID we store a list of all mailbox:UID[bodypart] pairs where this content occurs in. For search, we keep track of the indexed messages by GUID, so we can avoid reindexing duplicate mails. To return a search result, we can now map that GUID back to its mailbox:message pairs. That's why we need conversations.db for search. I can't help with upgrading from 2.4, unfortunately, but if you re-index your mailboxes once in conversations.db, you should be all set. Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 10:43:47 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 16:43:47 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Hi Sebastian, I'll answer below (and in green) your answers!! Thanks a lot for your explanations mate :) :) --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 15:57, Sebastian Hagedorn escribi?: > Hi, > > --On 7. Januar 2019 um 14:51:02 +0100 Egoitz Aurrekoetxea wrote: > >> This seems to take ages... > > why don't you run it for a single account first, to make sure that it actually helps? > > ====================================================================================================== > > YES :) IT HAS WORKED AS EXPECTED FOR A SINGLE ACCOUNT! HAVE LAUNCHED LATER THE : > > /USR/LOCAL/CYRUS/SBIN/CTL_CONVERSATIONSDB -R -Z > > (STILL RUNNING) > > AND LATER THE : > > /USR/LOCAL/CYRUS/SBIN/CTL_CONVERSATIONSDB -R -B > > ====================================================================================================== > >> I'm trying to figure the best way of >> implementing this and of clarifying concepts.... I'm running Squatter in >> rolling replication mode and exist the concept of conversations then. >> What is the exact role of each of them?. Squatter seems to index the >> mailbox but when something is not properly indexed instead of running >> Squatter you use ctl_conversations for reindexing some part again or... > > squatter is nowadays a bit of a misnomer, because it uses whatever index you have configured. In cyrus 2.4, squatter would always create a SQUAT index. When you run squatter with Xapian, it will build the index, but for the index to actually work, you also need the conversationsdb. squatter does not touch the conversationsdb! The index is only a pointer to the conversationsdb, not to actual messages. > > ====================================================================================================== > > I SEE!! THIS WAS THE KEY POINT I WAS LOOSING OR WHICH I WAS NOT ABLE TO SEE HOW ALL OF THEM WHERE RELATED... > > ====================================================================================================== > > Robert can probably explain this much better than me, but I think the problem is the following: > > * when you have conversations enabled in imapd.conf, normal deliveries to the mailboxes (e.g. using lmtpd) will update the conversationsdb > > * syncing (at least using the "old" mechanism, not sure about sync between instances running 3.0) does *not* update the conversationsdb > > ====================================================================================================== > > ALTHOUGH I ASSUME, HERE IF YOU RUN THE SQUATTER IN ROLLING MODE WITHIN A SLAVE SERVER SEEMS TO UPDATE THE INDEX... ALTHOUGH NOT SURE IF IT COULD DO TOO WITH CONVERSATIONSDB.. HOW DOES A SLAVE BEHAVE IN THIS SENSE?. COULD ANYONE HELP US PLEASE :) ?? > > ====================================================================================================== > > Once you have a running 3.0 server, you probably won't need to run ctl_conversationsdb ever again. But when you are at the stage of syncing mail from 2.4 to 3.0, you *will* need to rebuild each user's conversationdb at least once, after you have finished with syncing that user. > > ====================================================================================================== > > THIS IS IMPORTANT YES... > > ====================================================================================================== > > Again, this is all based on my understanding and not an official answer. > > ====================================================================================================== > > ANY I THANK A LOT ALL YOUR HELP!! MANY MANY THANKS MATE!! > > CHEERS!! > > ====================================================================================================== > > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > > That sounds like the conversationsdb issue I was talking about. Have you > tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER Mit freundlichen Gr??en Sebastian Hagedorn Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Mon Jan 7 10:48:50 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Mon, 07 Jan 2019 16:48:50 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: <1546876130.339157.1627790112.46C3834C@webmail.messagingengine.com> On Mon, Jan 7, 2019, at 4:13 PM, El?as Halld?r ?g?stsson wrote: > Regarding indexing and searching in body of emails; what if the body > text is encoded in base64 or quoted-printable? It won't yield any > unencoded search strings, or what? If the MIME body part is of type text, then base64 and QP-encoded bodies will get decoded for the search index. And they will get decoded before presenting the search result in snippets. Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 11:54:36 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 17:54:36 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> Message-ID: <1465831e8056cb640a8e09512205f156@sarenet.es> Hi Robert! Thank you so much for helping us (mainly which is the one boring the list with questions :) although I promise I've checked the doc before asking :) :) ). When you have a master/slave config... in the slave one, when running Squatter in rolling mode... does it update the conversations db too?. By the way, Squatter in rolling mode only makes sense in slave machines isn't it?. Many thanks! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 16:42, Robert Stepanek escribi?: > Hi, > > Sebastian is right: > > On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: > >> squatter is nowadays a bit of a misnomer, because it uses whatever index >> you have configured. In cyrus 2.4, squatter would always create a SQUAT >> index. When you run squatter with Xapian, it will build the index, but for >> the index to actually work, you also need the conversationsdb. > > conversations.db is indeed a misnomer now. The database was only used to keep track of mail threads (hence the name), but its role expanded. One of the indexes it stores is the SHA1 hashes of every message, and separate hashes for each of that message MIME parts. Such a hash is named the GUID, and for each GUID we store a list of all mailbox:UID[bodypart] pairs where this content occurs in. > > For search, we keep track of the indexed messages by GUID, so we can avoid reindexing duplicate mails. To return a search result, we can now map that GUID back to its mailbox:message pairs. That's why we need conversations.db for search. > > I can't help with upgrading from 2.4, unfortunately, but if you re-index your mailboxes once in conversations.db, you should be all set. > > Cheers, > Robert > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.menge at zdv.uni-tuebingen.de Mon Jan 7 11:58:01 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Mon, 07 Jan 2019 17:58:01 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> Message-ID: <20190107175801.Horde.pxgtWBjvSs4HBYXKkn6XfBj@webmail.uni-tuebingen.de> Hi, Quoting Egoitz Aurrekoetxea : > And, by the way.... > > when using Squatter instead of Xapian as a search engine.... what do we > really lost?. Just the fact of having a statistical worse results?. Is > it Xapian faster than squat engine?. > I didn't test Xapian but the Squatter search index is not used in cyrus-imapd 3.0 See https://github.com/cyrusimap/cyrus-imapd/issues/2598 > Sorry for having so many questions but... I suppose I don't have the > implications of each one totally clear :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario > hacerlo. > > El 07-01-2019 14:51, Egoitz Aurrekoetxea escribi?: > >> Hi mate! >> >> This seems to take ages... I'm trying to figure the best way of >> implementing this and of clarifying concepts.... I'm running >> Squatter in rolling replication mode and exist the concept of >> conversations then. What is the exact role of each of them?. >> Squatter seems to index the mailbox but when something is not >> properly indexed instead of running Squatter you use >> ctl_conversations for reindexing some part again or... >> >> Thanks a lot!! >> >> --- >> >> EGOITZ AURREKOETXEA >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol?gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es >> www.sarenet.es [1] >> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. >> >> El 07-01-2019 10:19, Sebastian Hagedorn escribi?: >> That sounds like the conversationsdb issue I was talking about. >> Have you tried these steps? >> >> ctl_conversationsdb -z USER >> ctl_conversationsdb -b USER >> >> I have been testing Xapian searches. Have seen, it's not able to find >> strings inside the body of the email. If I set in imap.conf >> "search_fuzzy_always: 1" no content is displayed in the searches of a >> Roundcube stock webmail. If I remove that config value from imap.conf >> and restart services, then search results appear. Does Xapian not index >> the body of emails?. Does Xapian, just index the headers?. But this >> affirmation does not seem to be possible in my case too... as I have in >> the config "search_index_headers: no". >> >> I'm using the following config : >> >> conversations: 1 >> search_engine: xapian >> search_index_headers: no >> search_batchsize: 8192 >> defaultsearchtier: t1 >> t1searchpartition-default: /expert/search >> t1searchpartition-expert2: /expert2/search >> t1searchpartition-expert3: /expert3/search >> >> Could anyone help me mates?. >> >> -- >> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 >> Regionales Rechenzentrum (RRZK) >> Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 > > > Links: > ------ > [1] http://www.sarenet.es -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From egoitz at sarenet.es Mon Jan 7 13:04:20 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 19:04:20 +0100 Subject: Question about manual replication (-u ) In-Reply-To: <9797216371bb00cdb59351370d5607d2@sarenet.es> References: <9797216371bb00cdb59351370d5607d2@sarenet.es> Message-ID: <878fd300fe5be3b58e712595106368b9@sarenet.es> Good afternoon, I know it seems a pretty stupid question, but some time ago, you cannot have a Cyrus server acting for instance as a master and as slave... it was not supported... worked... but not supported... so having multiple sync_client instances... perhaps could damage something (although I suppose not).... I'll probably have the same dount for ctl_conversations -z and -b multiple parallel commands... I know it seems ridiculous... but I ask it because I prefer to try get some knowledge from Cyrus gurus... :) :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 03-01-2019 16:32, Egoitz Aurrekoetxea escribi?: > Good afternoon, > > Is it possible to launch several instances of "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. Doing it just one mailbox at a time takes ages.... It would help me a lot, the fact of parallelizing and have no disk bottleneck issues.... > > I think it should be possible... isn't it?. Perhaps it just allowed between same version in source and dest?. Or can be done for instance too, with a 2.4 as master to 3.0 slave?. > > Cheers!! > > -- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Mon Jan 7 13:24:22 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Mon, 07 Jan 2019 19:24:22 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <1465831e8056cb640a8e09512205f156@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> <1465831e8056cb640a8e09512205f156@sarenet.es> Message-ID: <1546885462.2943983.1627952800.2CC6B3EA@webmail.messagingengine.com> Hi Egon, Yes, the slave should index in conversations.db automatically AFAIK. You should run squatter in rolling mode on the master, too. BTW: in 2014, Bron wrote a blog post about the search setup at FastMail: https://fastmail.blog/2014/12/01/email-search-system/It?s quite technical, but should give you a good idea at how it?s set up for fast indexing and search Cheers, Robert On Mon, Jan 7, 2019, at 5:54 PM, Egoitz Aurrekoetxea wrote: > Hi Robert! > > Thank you so much for helping us (mainly which is the one boring the > list with questions :) although I promise I've checked the doc before > asking :) :) ).> > When you have a master/slave config... in the slave one, when running > Squatter in rolling mode... does it update the conversations db too?. > By the way, Squatter in rolling mode only makes sense in slave > machines isn't it?.> > Many thanks! > > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario > hacerlo.> > El 07-01-2019 16:42, Robert Stepanek escribi?: >> Hi, >> >> Sebastian is right: >> >> On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: >>> >>> squatter is nowadays a bit of a misnomer, because it uses >>> whatever index>>> you have configured. In cyrus 2.4, squatter would always create >>> a SQUAT>>> index. When you run squatter with Xapian, it will build the index, >>> but for>>> the index to actually work, you also need the conversationsdb. >> >> conversations.db is indeed a misnomer now. The database was only used >> to keep track of mail threads (hence the name), but its role >> expanded. One of the indexes it stores is the SHA1 hashes of every >> message, and separate hashes for each of that message MIME parts. >> Such a hash is named the GUID, and for each GUID we store a list of >> all mailbox:UID[bodypart] pairs where this content occurs in.>> >> For search, we keep track of the indexed messages by GUID, so we can >> avoid reindexing duplicate mails. To return a search result, we can >> now map that GUID back to its mailbox:message pairs. That's why we >> need conversations.db for search.>> >> I can't help with upgrading from 2.4, unfortunately, but if you re- >> index your mailboxes once in conversations.db, you should be all set.>> >> Cheers, >> Robert >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 13:32:40 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 19:32:40 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <1546885462.2943983.1627952800.2CC6B3EA@webmail.messagingengine.com> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> <1465831e8056cb640a8e09512205f156@sarenet.es> <1546885462.2943983.1627952800.2CC6B3EA@webmail.messagingengine.com> Message-ID: Hi Robert! I see! I though you only needed to run Squatter in rolling mode in the slave.. I though the roling mode was just for slaves to take the Squatter changes caused by a normal squatter command launched in the master... So, if I run Squatter in the master in rolling mode... then I assume there's no need to launch manually squatter command in the master... isn't it?. I'm planning to upgrade some 2.3 running machines to either 2.4 or 3.0.... and you know... is a big responsability... am doing tons of sandbox testing... is it known (as Michael stated) not to be working traditional squatter in 3.0 if you don't want to use now the Xapian engine?. I'll read the post right now :) :) Thank you so much for all your help mate :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 19:24, Robert Stepanek escribi?: > Hi Egon, > > Yes, the slave should index in conversations.db automatically AFAIK. > > You should run squatter in rolling mode on the master, too. > > BTW: in 2014, Bron wrote a blog post about the search setup at FastMail: https://fastmail.blog/2014/12/01/email-search-system/ > It's quite technical, but should give you a good idea at how it's set up for fast indexing and search > > Cheers, Robert > > On Mon, Jan 7, 2019, at 5:54 PM, Egoitz Aurrekoetxea wrote: > > Hi Robert! > > Thank you so much for helping us (mainly which is the one boring the list with questions :) although I promise I've checked the doc before asking :) :) ). > > When you have a master/slave config... in the slave one, when running Squatter in rolling mode... does it update the conversations db too?. By the way, Squatter in rolling mode only makes sense in slave machines isn't it?. > > Many thanks! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 16:42, Robert Stepanek escribi?: > Hi, > > Sebastian is right: > > On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: > > squatter is nowadays a bit of a misnomer, because it uses whatever index > you have configured. In cyrus 2.4, squatter would always create a SQUAT > index. When you run squatter with Xapian, it will build the index, but for > the index to actually work, you also need the conversationsdb. > > conversations.db is indeed a misnomer now. The database was only used to keep track of mail threads (hence the name), but its role expanded. One of the indexes it stores is the SHA1 hashes of every message, and separate hashes for each of that message MIME parts. Such a hash is named the GUID, and for each GUID we store a list of all mailbox:UID[bodypart] pairs where this content occurs in. > > For search, we keep track of the indexed messages by GUID, so we can avoid reindexing duplicate mails. To return a search result, we can now map that GUID back to its mailbox:message pairs. That's why we need conversations.db for search. > > I can't help with upgrading from 2.4, unfortunately, but if you re-index your mailboxes once in conversations.db, you should be all set. > > Cheers, > Robert > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Tue Jan 8 03:26:14 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Tue, 08 Jan 2019 09:26:14 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> <1465831e8056cb640a8e09512205f156@sarenet.es> <1546885462.2943983.1627952800.2CC6B3EA@webmail.messagingengine.com> Message-ID: <1546935974.1208336.1628531736.2C101B2A@webmail.messagingengine.com> Hi, On Mon, Jan 7, 2019, at 7:32 PM, Egoitz Aurrekoetxea wrote: > So, if I run Squatter in the master in rolling mode... then I assume > there's no need to launch manually squatter command in the master... > isn't it?. If you indexed all messages in the mailbox before, then there shouldn't be a need to manually run squatter afterwards. The rolling squatter should pick up and index all newly created messages. > I'm planning to upgrade some 2.3 running machines to either 2.4 or > 3.0.... and you know... is a big responsability... am doing tons of > sandbox testing... is it known (as Michael stated) not to be working > traditional squatter in 3.0 if you don't want to use now the Xapian > engine?. Sorry, I have no experience with the upgrades from version 2. Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Jan 8 03:56:03 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 08 Jan 2019 09:56:03 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <1546935974.1208336.1628531736.2C101B2A@webmail.messagingengine.com> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <1546875768.333024.1627779704.6DEE718F@webmail.messagingengine.com> <1465831e8056cb640a8e09512205f156@sarenet.es> <1546885462.2943983.1627952800.2CC6B3EA@webmail.messagingengine.com> <1546935974.1208336.1628531736.2C101B2A@webmail.messagingengine.com> Message-ID: <9bd7712561eadeebc59c41717d918fe6@sarenet.es> Hi Robert! Thank you so much mate :) :) Best regards, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 08-01-2019 09:26, Robert Stepanek escribi?: > Hi, > > On Mon, Jan 7, 2019, at 7:32 PM, Egoitz Aurrekoetxea wrote: > >> So, if I run Squatter in the master in rolling mode... then I assume there's no need to launch manually squatter command in the master... isn't it?. > > If you indexed all messages in the mailbox before, then there shouldn't be a need to manually run squatter afterwards. The rolling squatter should pick up and index all newly created messages. > >> I'm planning to upgrade some 2.3 running machines to either 2.4 or 3.0.... and you know... is a big responsability... am doing tons of sandbox testing... is it known (as Michael stated) not to be working traditional squatter in 3.0 if you don't want to use now the Xapian engine?. > > Sorry, I have no experience with the upgrades from version 2. > > Cheers, > Robert Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Tue Jan 8 06:02:13 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Tue, 08 Jan 2019 06:02:13 -0500 Subject: Question about manual replication (-u ) In-Reply-To: <878fd300fe5be3b58e712595106368b9@sarenet.es> References: <9797216371bb00cdb59351370d5607d2@sarenet.es> <878fd300fe5be3b58e712595106368b9@sarenet.es> Message-ID: <3ebc8988-80da-453e-9bf3-ee4495926811@www.fastmail.com> Yep, that's totally safe. Even doing the same user twice at the same time should be safe, though it may do extra work. Bron. On Tue, Jan 8, 2019, at 05:05, Egoitz Aurrekoetxea wrote: > Good afternoon, > > I know it seems a pretty stupid question, but some time ago, you cannot have a Cyrus server acting for instance as a master and as slave... it was not supported... worked... but not supported... so having multiple sync_client instances... perhaps could damage something (although I suppose not).... I'll probably have the same dount for ctl_conversations -z and -b multiple parallel commands... > > I know it seems ridiculous... but I ask it because I prefer to try get some knowledge from Cyrus gurus... :) :) > > Cheers! > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 03-01-2019 16:32, Egoitz Aurrekoetxea escribi?: >> Good afternoon, >> >> Is it possible to launch several instances of "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. Doing it just one mailbox at a time takes ages.... It would help me a lot, the fact of parallelizing and have no disk bottleneck issues.... >> >> I think it should be possible... isn't it?. Perhaps it just allowed between same version in source and dest?. Or can be done for instance too, with a 2.4 as master to 3.0 slave?. >> >> Cheers!! >> -- >> >> sarenet >> *Egoitz Aurrekoetxea* >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol?gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es >> www.sarenet.es >> >> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Jan 8 06:28:27 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 08 Jan 2019 12:28:27 +0100 Subject: Question about manual replication (-u ) In-Reply-To: <3ebc8988-80da-453e-9bf3-ee4495926811@www.fastmail.com> References: <9797216371bb00cdb59351370d5607d2@sarenet.es> <878fd300fe5be3b58e712595106368b9@sarenet.es> <3ebc8988-80da-453e-9bf3-ee4495926811@www.fastmail.com> Message-ID: Thanks a lot Bron!!! :) :) --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 08-01-2019 12:02, Bron Gondwana escribi?: > Yep, that's totally safe. Even doing the same user twice at the same time should be safe, though it may do extra work. > > Bron. > > On Tue, Jan 8, 2019, at 05:05, Egoitz Aurrekoetxea wrote: > > Good afternoon, > > I know it seems a pretty stupid question, but some time ago, you cannot have a Cyrus server acting for instance as a master and as slave... it was not supported... worked... but not supported... so having multiple sync_client instances... perhaps could damage something (although I suppose not).... I'll probably have the same dount for ctl_conversations -z and -b multiple parallel commands... > > I know it seems ridiculous... but I ask it because I prefer to try get some knowledge from Cyrus gurus... :) :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 03-01-2019 16:32, Egoitz Aurrekoetxea escribi?: > > Good afternoon, > > Is it possible to launch several instances of "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. Doing it just one mailbox at a time takes ages.... It would help me a lot, the fact of parallelizing and have no disk bottleneck issues.... > > I think it should be possible... isn't it?. Perhaps it just allowed between same version in source and dest?. Or can be done for instance too, with a 2.4 as master to 3.0 slave?. > > Cheers!! > > -- > > EGOITZ AURREKOETXEA > Departamento de sistemas > > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Jan 8 12:40:17 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 08 Jan 2019 18:40:17 +0100 Subject: Help with few questions Message-ID: <94ac4fc1e15fba426ad93f65ca34dbc5@sarenet.es> Good afternoon, I would be very thankful if you could clarify me these questions. Have not been able to find documentation for them and my testing env has not been able too :) of clarifying me these doubts... Here I go :) - Some friendly mate, has told me here that when you enable Xapian engine in Cyrus 3.0 and have a master/slave replication, Squatter in rolling mode should be enabled in both, the master and the slave. I understand the master should be using it for indexing with Xapian the new mail entering in mailboxes, mail movement... and in the slave for applying the Xapian transactions being applied and sent from the master. But, how can be that having the same line in events, depending in whether it's master or slave does what just should do?. The sync_log: on and sync_log_channels: squatter should be too enabled in both master or slave?. Or just in one of them?. I haven't never run a Cyrus replication of Xapian databases.. so this is new for me and don't understand how all this param behave and if should be enabled in both master and slave. I would like to use rolling Squatter in the master and that changes to be sent to the slave. - search_fuzzy_always: 1 , causes all searches to go through Xapian engine. Being so good as it seems (and the way it speeds in my testing env search operations, it's nice!!), what could be the reason for not having it enabled by default?. Can it have some kind of problem?. I can't see them... Just for avoiding surprises. - Bron, in this post https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail was not able to handle with just one Xapian default database (even on SSD disks) all traffic. So he said Fastmail was using a in-memory filesystem for a database (called temp) for new email. Later another for cleaning up that in memory filesystem. And later one more, for keeping definitively the content. You seemed to use a Squatter command for moving elements between databases. Concretely (sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong). I assume that compacts all elements from all databases to archive?. If I wanted to compact elements from temp to data, the command should be "sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data -t temp -u brong" (in this example for user brong) ??. I assume you launch something like it with a cron weekly and it's done?. - If something went wrong when the upgrade proccess from 2.4 to 3.0, could I setup 3.0 as master of the 2.4 and later make 2.4 master again?. Could that cause info loosing?. I assume yes but just for knowing posibilities. - When a Xapian or conversation index becomes broken, a reconstruct could recover?. What could be the repairing procedure?. Thank you so much mates. Sorry for having so many questions... but neither testing nor documentation or mailing lists have been able to clarify me them. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From byrnejb at gmx.com Tue Jan 8 14:12:00 2019 From: byrnejb at gmx.com (James B Byrne) Date: Tue, 8 Jan 2019 20:12:00 +0100 Subject: upgrade to cyrus_imap or saslauth or both gon horribly wrong Message-ID: FreeBSD-11.2p7 cyrus-imapd30-3.0.8_2 cyrus-sasl-saslauthd-2.1.27 cyrus-sasl-2.1.27 This morning we upgraded our cyrus_imap server using the FreeBSD pkg package manager. Following this we are unable to authenticate with imap. The error we receive is this: Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to saslauthd server: Permission denied Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername [server address] plaintext username SASL(-1): generic failure: checkpass failed imapd.conf was not changed. it contains this: sasl_mech_list: PLAIN sasl_pwcheck_method: saslauthd I am posting this from a temporary email because, duhh, I cannot access my regular mailbox. I am open to any reasonable suggestions as to how to fix this, quickly. From dwhite at olp.net Tue Jan 8 14:24:15 2019 From: dwhite at olp.net (Dan White) Date: Tue, 8 Jan 2019 13:24:15 -0600 Subject: upgrade to cyrus_imap or saslauth or both gon horribly wrong In-Reply-To: References: Message-ID: <20190108192415.GI2172@dan.olp.net> On 01/08/19?20:12?+0100, James B Byrne wrote: >FreeBSD-11.2p7 >cyrus-imapd30-3.0.8_2 >cyrus-sasl-saslauthd-2.1.27 >cyrus-sasl-2.1.27 > >This morning we upgraded our cyrus_imap server using the FreeBSD pkg package manager. Following this we are unable to authenticate with imap. The error we receive is this: > >Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to saslauthd server: Permission denied Find the location of your saslauthd mux (unix domain socket) within the filesystem and verify the permissions of its path (typically somewhere underneath /var). It should allow access to the cyrus user. You can use testsaslauthd, as the cyrus user, to verify permissions. >Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername [server address] plaintext username SASL(-1): generic failure: checkpass failed > >imapd.conf was not changed. it contains this: > >sasl_mech_list: PLAIN >sasl_pwcheck_method: saslauthd > > >I am posting this from a temporary email because, duhh, I cannot access my regular mailbox. > >I am open to any reasonable suggestions as to how to fix this, quickly. From byrnejb at gmx.com Tue Jan 8 14:29:00 2019 From: byrnejb at gmx.com (James B Byrne) Date: Tue, 8 Jan 2019 20:29:00 +0100 Subject: upgrade to cyrus_imap or saslauth or both gon horribly wrong In-Reply-To: References: Message-ID: > FreeBSD-11.2p7 > cyrus-imapd30-3.0.8_2 > cyrus-sasl-saslauthd-2.1.27 > cyrus-sasl-2.1.27 > > This morning we upgraded our cyrus_imap server using the FreeBSD pkg package manager. Following this we are unable to authenticate with imap. The error we receive is this: > > Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to saslauthd server: Permission denied > Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername [server address] plaintext username SASL(-1): generic failure: checkpass failed > > imapd.conf was not changed. it contains this: > > sasl_mech_list: PLAIN > sasl_pwcheck_method: saslauthd > > > I am posting this from a temporary email because, duhh, I cannot access my regular mailbox. > > I am open to any reasonable suggestions as to how to fix this, quickly. However, if I do this then I get a success. read -ers -p 'prompt: ' PASSWD ; echo -e '\n' prompt: read -ers -p 'prompt: ' USERNAME ; echo -e '\n' prompt: testsaslauthd -u $USERNAME -p $PASSWD 0: OK "Success." Does anyone here know what is going on or how to fix it? From michael.menge at zdv.uni-tuebingen.de Wed Jan 9 04:29:07 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 09 Jan 2019 10:29:07 +0100 Subject: upgrade to cyrus_imap or saslauth or both gon horribly wrong In-Reply-To: References: Message-ID: <20190109102907.Horde.r4yHRsPqt7jBf146Za3DZTf@webmail.uni-tuebingen.de> Hi, Quoting James B Byrne : >> FreeBSD-11.2p7 >> cyrus-imapd30-3.0.8_2 >> cyrus-sasl-saslauthd-2.1.27 >> cyrus-sasl-2.1.27 >> >> This morning we upgraded our cyrus_imap server using the FreeBSD >> pkg package manager. Following this we are unable to authenticate >> with imap. The error we receive is this: >> >> Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to >> saslauthd server: Permission denied >> Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername >> [server address] plaintext username SASL(-1): generic failure: >> checkpass failed >> >> imapd.conf was not changed. it contains this: >> >> sasl_mech_list: PLAIN >> sasl_pwcheck_method: saslauthd >> >> >> I am posting this from a temporary email because, duhh, I cannot >> access my regular mailbox. >> >> I am open to any reasonable suggestions as to how to fix this, quickly. > > However, if I do this then I get a success. > > read -ers -p 'prompt: ' PASSWD ; echo -e '\n' > prompt: > > read -ers -p 'prompt: ' USERNAME ; echo -e '\n' > prompt: > > testsaslauthd -u $USERNAME -p $PASSWD > 0: OK "Success." > > Does anyone here know what is going on or how to fix it? are you using SELinux in enforced mode? If yes check your SELinux audit log (/var/log/audit/audit.log) > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From Hagedorn at uni-koeln.de Wed Jan 9 06:59:27 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 09 Jan 2019 12:59:27 +0100 Subject: Help with few questions In-Reply-To: <94ac4fc1e15fba426ad93f65ca34dbc5@sarenet.es> References: <94ac4fc1e15fba426ad93f65ca34dbc5@sarenet.es> Message-ID: <29234C3964AC9CFE8FD7E2F4@tyrion.rrz.uni-koeln.de> Hi, I will try to answer what I can ... see below. --On 8. Januar 2019 um 18:40:17 +0100 Egoitz Aurrekoetxea wrote: > - search_fuzzy_always: 1 , causes all searches to go through Xapian > engine. Being so good as it seems (and the way it speeds in my testing > env search operations, it's nice!!), what could be the reason for not > having it enabled by default?. Can it have some kind of problem?. I > can't see them... Just for avoiding surprises. With FUZZY you may get more matches than without. Look here for an explanation: The example in 3. is a good one: 3. The FUZZY Search Key The FUZZY search key takes another search key as its argument. The server is allowed to perform all matching in an implementation- defined manner for this search key, including ignoring the active comparator as defined by [RFC5255]. Typically, this would be used to search for strings. For example: C: A1 SEARCH FUZZY (SUBJECT "IMAP break") S: * SEARCH 1 5 10 S: A1 OK Search completed. Besides matching messages with a subject of "IMAP break", the above search may also match messages with subjects "broken IMAP", "IMAP is broken", or anything else the server decides that might be a good match. Note that the *server* decides what "might be a good match". When all searches become FUZZY it might confuse users, but on the other hand I doubt there is a single IMAP client that lets the user choose whether a particular search should be FUZZY or not ... > - Bron, in this post > https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail > was not able to handle with just one Xapian default database (even on > SSD disks) all traffic. So he said Fastmail was using a in-memory > filesystem for a database (called temp) for new email. Later another for > cleaning up that in memory filesystem. And later one more, for keeping > definitively the content. You seemed to use a Squatter command for > moving elements between databases. Concretely (sudo -u cyrus > /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z > archive -t temp,meta,data,archive -u brong). I assume that compacts all > elements from all databases to archive?. If I wanted to compact elements > from temp to data, the command should be "sudo -u cyrus > /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data > -t temp -u brong" (in this example for user brong) ??. I assume you > launch something like it with a cron weekly and it's done?. That depends entirely on your system. Without knowing the number of users you have, the number of mails that arrive per hour, what kind of storage systems you have, how much free RAM your servers have, it is impossible to say how often you need to squatter in compact mode. On a test server I set up I run a cron job every 2 minutes to check if tmpfs is getting tight ... > - If something went wrong when the upgrade proccess from 2.4 to 3.0, > could I setup 3.0 as master of the 2.4 and later make 2.4 master again?. > Could that cause info loosing?. I assume yes but just > for knowing posibilities. You have to describe in more detail what you mean. The point of no return is usually when you start to deliver new mail to the new server. You cannot sync from 3.0 to 2.4. That means if you start to deliver new mail to the 3.0 server, you can't go back to 2.4 without losing the messages that have arrived n the meantime. Strictly speaking to could manually copy the mailboxes from the 3.0 server to the 2.4 and run reconstruct, but I don't consider that a viable option. > - When a Xapian or conversation index becomes broken, a reconstruct > could recover?. What could be the repairing procedure?. ctl_conversationsdb -z "zaps" the conversationsdb ? it basically empties it. Then you can recreate it with ctl_conversationsdb -b. The "reconstruct" command does not touch the conversationsdb. If the actual Xapian index should be broken, I guess you'd have to delete the index files and run squatter again. Cheers Sebastian -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From egoitz at sarenet.es Wed Jan 9 07:37:02 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Wed, 09 Jan 2019 13:37:02 +0100 Subject: Help with few questions In-Reply-To: <29234C3964AC9CFE8FD7E2F4@tyrion.rrz.uni-koeln.de> References: <94ac4fc1e15fba426ad93f65ca34dbc5@sarenet.es> <29234C3964AC9CFE8FD7E2F4@tyrion.rrz.uni-koeln.de> Message-ID: <78a949ab416f9a191d410cfc9c19d686@sarenet.es> Hi Sebastian!! Thank you so much mate!!. Point 2 is the most uncertain for me. I answer your comments below :) Point 1 -> It's clear... yes I think so too... I was planning to force all searches as FUZZY with search_fuzzy_always: 1 as it seems really nice :) Point 2 -> So, you do a df every two minutes to see free space in tmpfs?... and when you see it starts growing you launch a "sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong" equivalent command?. You would launch it without -u I suppose, for doing Squatter for all users (no sense of doing it for just one user... isn't it?)... I assume this command would move all items from temp, meta and data to archive database. So, for instance if you launch a "sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data -t temp,meta,data -u brong" it moves (and then empties temp and meta) temp and meta to data and even compacts the own data database?. We usually have 2000 imap concurrent connections, 150 concurrent pop3 and receive 120 messages/min more or less... and SSD array of disks yes.. for each server... Point 3 -> Totally agree. Just for confirming. Point 4 -> If having a crash in a server... the Cyrus log file, would throw which mailboxes need a ctl_conversations -z and -b or "it's better" doing it for all accounts (with -r and without -u?)?. When talking about Xapian databases (not conversations) I assume some error would appear for them surely... the same way as for instance if cyrus.seen would be corrupt... isn't it?. Thanks a lot again. Your help is really really important for at this stage... Cheers!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 09-01-2019 12:59, Sebastian Hagedorn escribi?: > Hi, > > I will try to answer what I can ... see below. > > --On 8. Januar 2019 um 18:40:17 +0100 Egoitz Aurrekoetxea wrote: > >> - search_fuzzy_always: 1 , causes all searches to go through Xapian >> engine. Being so good as it seems (and the way it speeds in my testing >> env search operations, it's nice!!), what could be the reason for not >> having it enabled by default?. Can it have some kind of problem?. I >> can't see them... Just for avoiding surprises. > > With FUZZY you may get more matches than without. Look here for an explanation: > > > > The example in 3. is a good one: > > 3. The FUZZY Search Key > > The FUZZY search key takes another search key as its argument. The > server is allowed to perform all matching in an implementation- > defined manner for this search key, including ignoring the active > comparator as defined by [RFC5255]. Typically, this would be used to > search for strings. For example: > > C: A1 SEARCH FUZZY (SUBJECT "IMAP break") > S: * SEARCH 1 5 10 > S: A1 OK Search completed. > > Besides matching messages with a subject of "IMAP break", the above > search may also match messages with subjects "broken IMAP", "IMAP is > broken", or anything else the server decides that might be a good > match. > > Note that the *server* decides what "might be a good match". When all searches become FUZZY it might confuse users, but on the other hand I doubt there is a single IMAP client that lets the user choose whether a particular search should be FUZZY or not ... > >> - Bron, in this post >> https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail >> was not able to handle with just one Xapian default database (even on >> SSD disks) all traffic. So he said Fastmail was using a in-memory >> filesystem for a database (called temp) for new email. Later another for >> cleaning up that in memory filesystem. And later one more, for keeping >> definitively the content. You seemed to use a Squatter command for >> moving elements between databases. Concretely (sudo -u cyrus >> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z >> archive -t temp,meta,data,archive -u brong). I assume that compacts all >> elements from all databases to archive?. If I wanted to compact elements >> from temp to data, the command should be "sudo -u cyrus >> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data >> -t temp -u brong" (in this example for user brong) ??. I assume you >> launch something like it with a cron weekly and it's done?. > > That depends entirely on your system. Without knowing the number of users you have, the number of mails that arrive per hour, what kind of storage systems you have, how much free RAM your servers have, it is impossible to say how often you need to squatter in compact mode. On a test server I set up I run a cron job every 2 minutes to check if tmpfs is getting tight ... > >> - If something went wrong when the upgrade proccess from 2.4 to 3.0, >> could I setup 3.0 as master of the 2.4 and later make 2.4 master again?. >> Could that cause info loosing?. I assume yes but just >> for knowing posibilities. > > You have to describe in more detail what you mean. The point of no return is usually when you start to deliver new mail to the new server. You cannot sync from 3.0 to 2.4. That means if you start to deliver new mail to the 3.0 server, you can't go back to 2.4 without losing the messages that have arrived n the meantime. Strictly speaking to could manually copy the mailboxes from the 3.0 server to the 2.4 and run reconstruct, but I don't consider that a viable option. > >> - When a Xapian or conversation index becomes broken, a reconstruct >> could recover?. What could be the repairing procedure?. > > ctl_conversationsdb -z "zaps" the conversationsdb - it basically empties it. Then you can recreate it with ctl_conversationsdb -b. The "reconstruct" command does not touch the conversationsdb. If the actual Xapian index should be broken, I guess you'd have to delete the index files and run squatter again. > > Cheers > Sebastian Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From byrnejb at harte-lyne.ca Fri Jan 11 11:34:44 2019 From: byrnejb at harte-lyne.ca (James B. Byrne) Date: Fri, 11 Jan 2019 11:34:44 -0500 Subject: Upgrade to cyrus-imapd and connection drops when searching mailboxes Message-ID: cyrus-imapd30-3.0.8_2 Name : cyrus-imapd30 Version : 3.0.8_2 Installed on : Tue Jan 8 11:07:34 2019 EST Origin : mail/cyrus-imapd30 Architecture : FreeBSD:11:amd64 Prefix : /usr/local Categories : ipv6 mail Licenses : BSD4CLAUSE Maintainer : ume at FreeBSD.org Since upgrading to this version from 3.0.8_1 on January 8 we have seen a considerable increase of messages in the maillog that look similar to this: CYRUS/lmtpunix[79768]: mailbox: longlock user.name.delivery for 1.4 seconds Where the number of seconds varies between 1.x and 11.x. When I say considerably I mean from between 10 and 30 per log rollover to over 800 today and it is not yet noon. Perhaps coincidently, users are reporting imap connections dropping when they are performing text searches on their mailboxes. The user mail client being squirrelmail. Are the longlock reports and the imap connection drop related? If imap drops a connection is the reason for that event logged? Can it be logged? -- *** e-Mail is NOT a SECURE channel *** Do NOT transmit sensitive data via e-Mail Do NOT open attachments nor follow links sent by e-Mail James B. Byrne mailto:ByrneJB at Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3 From michael.menge at zdv.uni-tuebingen.de Fri Jan 11 17:51:41 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Fri, 11 Jan 2019 23:51:41 +0100 Subject: Upgrade to cyrus-imapd and connection drops when searching mailboxes In-Reply-To: Message-ID: <20190111235141.Horde.bIX3Bbj7G0ncRkwE6JGKZy6@webmail.uni-tuebingen.de> Hi, Quoting "James B. Byrne via Info-cyrus" : > cyrus-imapd30-3.0.8_2 > Name : cyrus-imapd30 > Version : 3.0.8_2 > Installed on : Tue Jan 8 11:07:34 2019 EST > Origin : mail/cyrus-imapd30 > Architecture : FreeBSD:11:amd64 > Prefix : /usr/local > Categories : ipv6 mail > Licenses : BSD4CLAUSE > Maintainer : ume at FreeBSD.org > > Since upgrading to this version from 3.0.8_1 on January 8 we have seen > a considerable increase of messages in the maillog that look similar > to this: > > CYRUS/lmtpunix[79768]: mailbox: longlock user.name.delivery for 1.4 > seconds > > Where the number of seconds varies between 1.x and 11.x. When I say > considerably I mean from between 10 and 30 per log rollover to over > 800 today and it is not yet noon. > > Perhaps coincidently, users are reporting imap connections dropping > when they are performing text searches on their mailboxes. The user > mail client being squirrelmail. > have you configured a search engine? and have you enabled the conversation db. Xapian and Squatter seam to require enabled conversation db to use the search index. At least for squatter there is even a performance regression for TEXT search even if conversation db is enabled (https://github.com/cyrusimap/cyrus-imapd/issues/2598) > Are the longlock reports and the imap connection drop related? > An cyrus processes tires to gain access to the mailbox while an other process is still accessing it. If this takes to long the client might disconnect. You can try telemetry logging to discover what is happening . > If imap drops a connection is the reason for that event logged? Can > it be logged? > The question is which side drops the connection the cyrus-imapd process or the imap client (squirrelmail)? tcpdum and telemetry logging might reveal the answer. What timeout is configured in squirrelmail? -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From byrnejb at harte-lyne.ca Sat Jan 12 08:47:25 2019 From: byrnejb at harte-lyne.ca (James B. Byrne) Date: Sat, 12 Jan 2019 08:47:25 -0500 Subject: Upgrade to cyrus-imapd and connection drops when searching mailboxes In-Reply-To: <20190111235141.Horde.bIX3Bbj7G0ncRkwE6JGKZy6@webmail.uni-tuebingen.de> References: <20190111235141.Horde.bIX3Bbj7G0ncRkwE6JGKZy6@webmail.uni-tuebingen.de> Message-ID: On Fri, January 11, 2019 17:51, Michael Menge wrote: Thank you for your assistance. > > have you configured a search engine? and have you enabled the > conversation db. > Xapian and Squatter seam to require enabled conversation db to use the > search index. At least for squatter there is even a performance > regression for TEXT search even if conversation db is enabled > (https://github.com/cyrusimap/cyrus-imapd/issues/2598) I do not believe that I have configured a search engine. > > >> Are the longlock reports and the imap connection drop related? >> > An cyrus processes tires to gain access to the mailbox while an > other process is still accessing it. If this takes to long the client > might disconnect. You can try telemetry logging to discover what is > happening. >> If imap drops a connection is the reason for that event logged? Can >> it be logged? >> > The question is which side drops the connection the cyrus-imapd > process or the imap client (squirrelmail)? tcpdum and telemetry > logging might reveal the answer. Squirrelmail reports that the IMAP server dropped the connection as part of the error message. In any case, even if it is the squrrelmail client that is dropping the connection one would expect that imap service could report this event. So the question remains, how does one log the cause of cyrus_imapd dropping a connection? As to the cause of the problem. It was an excessive load on the imap daemon from a persistent brute force attack. A recent reconfiguration and change of server host resulted in port 993 being opened to unrestricted public access. This attracted the usual assortment of script kiddies, security 'researchers' of various ilk, and so forth. Closing that port down immediately resolved the issue. Thanks for the suggestions. I will look into these now that the crisis has passed. Regards, -- *** e-Mail is NOT a SECURE channel *** Do NOT transmit sensitive data via e-Mail Do NOT open attachments nor follow links sent by e-Mail James B. Byrne mailto:ByrneJB at Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3 From ume at mahoroba.org Sat Jan 12 19:21:07 2019 From: ume at mahoroba.org (Hajimu UMEMOTO) Date: Sun, 13 Jan 2019 09:21:07 +0900 Subject: upgrade to cyrus_imap or saslauth or both gon horribly wrong In-Reply-To: References: Message-ID: Hi, >>>>> On Tue, 8 Jan 2019 20:12:00 +0100 >>>>> "James B Byrne" said: byrnejb> FreeBSD-11.2p7 byrnejb> cyrus-imapd30-3.0.8_2 byrnejb> cyrus-sasl-saslauthd-2.1.27 byrnejb> cyrus-sasl-2.1.27 byrnejb> This morning we upgraded our cyrus_imap server using the FreeBSD pkg package manager. Following this we are unable to authenticate with imap. The error we receive is this: byrnejb> Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to saslauthd server: Permission denied byrnejb> Jan 8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername [server address] plaintext username SASL(-1): generic failure: checkpass failed It seems the recent pkg doesn't set the owner of /var/run/saslauthd correctly. So, I committed to set it explicitly. Please try cyrus-sasl-saslauthd-2.1.27_1. Sincerely, -- Hajimu UMEMOTO ume at mahoroba.org ume at FreeBSD.org http://www.mahoroba.org/~ume/ From falon at ruparpiemonte.it Mon Jan 14 06:43:37 2019 From: falon at ruparpiemonte.it (Marco) Date: Mon, 14 Jan 2019 12:43:37 +0100 Subject: Cyrus tools and Splunk Message-ID: Hello, I have written some tools for Cyrus, in particular an app for Splunk. It's designed for my environment and it is at this time poorly documented, ...but you could find something of your interest. So I decided to share this work. https://splunkbase.splunk.com/app/4345/ Let me know if contribution like this could violate some guideline of Cyrus IMAP. Thank you. Have a nice day. Marco From daniel-listas at gmx.net Tue Jan 15 11:33:29 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Tue, 15 Jan 2019 13:33:29 -0300 Subject: cyrus-imapd not starting after upgrade Message-ID: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net> Hi all! After quite some time, today I decided to update the mail server from Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3). All without problems until I reach the part of cyrus-imapd that does not start. This is what I see in the log: ---------- Jan 14 23:10:45 mail systemd[1]: Started Cyrus IMAP/POP3 daemons. Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: skiplist: clean shutdown file missing, updating recovery stamp Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: recovering cyrus databases Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: done recovering cyrus databases Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Repacking mailbox user.admin.TareasCron version 12 Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Expired 0 and expunged 0 out of 28809 messages from 80 mailboxes Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: pruning back 3.00 days Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: purged 0 out of 438 entries Jan 14 23:10:46 mail cyrus/tls_prune[5335]: twoskip: invalid magic header: /var/lib/cyrus/tls_sessions.db Jan 14 23:10:46 mail cyrus/tls_prune[5335]: cyrusdb: opening /var/lib/cyrus/tls_sessions.db with backend skiplist (requested twoskip) Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: recovered /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0 seconds Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: checkpointed /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0.091 sec Jan 14 23:10:46 mail cyrus/tls_prune[5335]: tls_prune: purged 2 out of 223 entries Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for service 'nntp' Jan 14 23:10:46 mail cyrus/master[5311]: exiting Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Main process exited, code=exited, status=78/n/a Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Unit entered failed state. Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Failed with result 'exit-code'. ---------- I'm not sure what the problem is but that "invalid magic header" makes me think that maybe it changed the header format of /var/lib/cyrus/tls_sessions.db and the migration process did not do the corresponding conversion. Can that be the reason why it doesn't start or I'm missing something else? Any ideas that can bring more light? The associated problem is that because of this it seems that Postfix can not deliver the mails since there is no /var/run/cyrus/socket/lmtp. Thanks in advance. Kind regards, Daniel From michael.menge at zdv.uni-tuebingen.de Tue Jan 15 11:54:15 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 15 Jan 2019 17:54:15 +0100 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net> Message-ID: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> Quoting Daniel Bareiro : > Hi all! > > After quite some time, today I decided to update the mail server from > Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3). > > All without problems until I reach the part of cyrus-imapd that does not > start. This is what I see in the log: > > ---------- > Jan 14 23:10:45 mail systemd[1]: Started Cyrus IMAP/POP3 daemons. > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: skiplist: clean shutdown > file missing, updating recovery stamp > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: recovering cyrus databases > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: done recovering cyrus > databases > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Repacking mailbox > user.admin.TareasCron version 12 > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Expired 0 and expunged 0 > out of 28809 messages from 80 mailboxes > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: pruning > back 3.00 days > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: purged 0 > out of 438 entries > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: twoskip: invalid magic > header: /var/lib/cyrus/tls_sessions.db > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: cyrusdb: opening > /var/lib/cyrus/tls_sessions.db with backend skiplist (requested twoskip) > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: recovered > /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0 seconds > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: checkpointed > /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0.091 sec > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: tls_prune: purged 2 out of > 223 entries > Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for > service 'nntp' > Jan 14 23:10:46 mail cyrus/master[5311]: exiting > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Main process > exited, code=exited, status=78/n/a > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Unit entered > failed state. > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Failed with result > 'exit-code'. > ---------- > > I'm not sure what the problem is but that "invalid magic header" makes > me think that maybe it changed the header format of > /var/lib/cyrus/tls_sessions.db and the migration process did not do the > corresponding conversion. Can that be the reason why it doesn't start or > I'm missing something else? Any ideas that can bring more light? > some defaults for database format have changed between 2.4 and 2.5 but this is automatically handled. You Problem lies in Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for service 'nntp' Jan 14 23:10:46 mail cyrus/master[5311]: exiting Either the executables are installed in the wrong place, or cyrus was compiled/packaged without news support. You can comment out nntp in /etc/cyrus.conf to try to start without running an nntp server > The associated problem is that because of this it seems that Postfix can > not deliver the mails since there is no /var/run/cyrus/socket/lmtp. As cyrus aborted the start there is no process listening -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From morgan at orst.edu Tue Jan 15 11:57:38 2019 From: morgan at orst.edu (Andrew Morgan) Date: Tue, 15 Jan 2019 08:57:38 -0800 (PST) Subject: cyrus-imapd not starting after upgrade In-Reply-To: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net> References: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net> Message-ID: On Tue, 15 Jan 2019, Daniel Bareiro wrote: > Hi all! > > After quite some time, today I decided to update the mail server from > Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3). > > All without problems until I reach the part of cyrus-imapd that does not > start. This is what I see in the log: > > ---------- > Jan 14 23:10:45 mail systemd[1]: Started Cyrus IMAP/POP3 daemons. > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: skiplist: clean shutdown > file missing, updating recovery stamp > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: recovering cyrus databases > Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: done recovering cyrus > databases > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Repacking mailbox > user.admin.TareasCron version 12 > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Expired 0 and expunged 0 > out of 28809 messages from 80 mailboxes > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: pruning > back 3.00 days > Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: purged 0 > out of 438 entries > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: twoskip: invalid magic > header: /var/lib/cyrus/tls_sessions.db > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: cyrusdb: opening > /var/lib/cyrus/tls_sessions.db with backend skiplist (requested twoskip) > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: recovered > /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0 seconds > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: checkpointed > /var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0.091 sec > Jan 14 23:10:46 mail cyrus/tls_prune[5335]: tls_prune: purged 2 out of > 223 entries > Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for > service 'nntp' > Jan 14 23:10:46 mail cyrus/master[5311]: exiting > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Main process > exited, code=exited, status=78/n/a > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Unit entered > failed state. > Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Failed with result > 'exit-code'. > ---------- > > I'm not sure what the problem is but that "invalid magic header" makes > me think that maybe it changed the header format of > /var/lib/cyrus/tls_sessions.db and the migration process did not do the > corresponding conversion. Can that be the reason why it doesn't start or > I'm missing something else? Any ideas that can bring more light? > > The associated problem is that because of this it seems that Postfix can > not deliver the mails since there is no /var/run/cyrus/socket/lmtp. It wants tls_sessions.db to be a twoskip-format file, but the current format is skiplist. However, it was able to detect this and open it as skiplist. You can fix this issue by stopping Cyrus, removing tls_sessions.db, and starting Cyrus. However, your real problem seems to be the missing nntp executable: Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for service 'nntp' Do you use NNTP? You could comment it out of cyrus.conf in order to get the rest of Cyrus up and running. Take a look at the release notes for v2.5.0: https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.0.html It covers important changes from v2.4 to v2.5. You may need to update your cyrus.conf and imapd.conf files. Thanks, Andy From daniel-listas at gmx.net Tue Jan 15 13:05:50 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Tue, 15 Jan 2019 15:05:50 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> Message-ID: <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> Hi, Michael. Thanks for your reply. On 15/1/19 13:54, Michael Menge wrote: >> I'm not sure what the problem is but that "invalid magic header" makes >> me think that maybe it changed the header format of >> /var/lib/cyrus/tls_sessions.db and the migration process did not do the >> corresponding conversion. Can that be the reason why it doesn't start or >> I'm missing something else? Any ideas that can bring more light? > some defaults for database format have changed between 2.4 and 2.5 but this > is automatically handled. Thanks for confirm. > You Problem lies in > > Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for > service 'nntp' > Jan 14 23:10:46 mail cyrus/master[5311]: exiting > > Either the executables are installed in the wrong place, or cyrus was > compiled/packaged without news support. You can comment out nntp in > /etc/cyrus.conf to try to start without running an nntp server It seems that support for NNTP was separated in Debian Stretch regarding to Debian Jessie. I don't use NNTP, so I simply commented out the "nntp" line in the /etc/cyrus.conf configuration file. I think I had tried that before, although it seems to me that time I forgot to do it also with the "nntps" line. Then I tried again to start the service and now the error changed: ---------- Jan 15 14:13:59 mail cyrus/master[20737]: cannot find executable for service 'http' ---------- But I'm not sure if this has been included in another separate package: ---------- root at mail:/var/run/cyrus/socket# aptitude search ^cyrus idA cyrus-admin - Cyrus mail system - administration tools c cyrus-admin-2.4 - p cyrus-caldav - Cyrus mail system - CalDAV and CardDAV support idA cyrus-clients - Cyrus mail system - test clients c cyrus-clients-2.4 - idA cyrus-common - Cyrus mail system - common files i cyrus-common-2.4 - Cyrus mail system - common files [dummy package] p cyrus-dev - Cyrus mail system - developer files p cyrus-doc - Cyrus mail system - documentation files idA cyrus-imapd - Cyrus mail system - IMAP support c cyrus-imapd-2.4 - p cyrus-imspd - Internet Message Support Protocol daemon p cyrus-murder - Cyrus mail system - proxies and aggregator p cyrus-nntpd - Cyrus mail system - NNTP support p cyrus-pop3d - Cyrus mail system - POP3 support p cyrus-replication - Cyrus mail system - replication p cyrus-sasl2-doc - Cyrus SASL - documentation ---------- The fact is that if I also comment out the "http" line, now it seems that I get a syntax error: ---------- Jan 15 14:14:28 mail cyrus/master[20778]: configuration file /etc/cyrus.conf: bad character '_' in name on line 101 Jan 15 14:14:28 mail cyrus/master[20778]: exiting ---------- I think the configuration file already came with the "squatter_a" line and I just removed the comment. Unless this is a bug in the configuration file provided by Stretch for Cyrus 2.5.10: ---------- 99 # reindex all mailboxes (fulltext) daily 100 # DGB - 20190114 101 squatter_a cmd="/usr/sbin/cyrus squatter" at=0517 ---------- >> The associated problem is that because of this it seems that Postfix can >> not deliver the mails since there is no /var/run/cyrus/socket/lmtp. > As cyrus aborted the start there is no process listening That's what I thought :-) Thanks for your time. Kind regards, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From daniel-listas at gmx.net Tue Jan 15 13:36:24 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Tue, 15 Jan 2019 15:36:24 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> Message-ID: On 15/1/19 15:05, Daniel Bareiro wrote: > The fact is that if I also comment out the "http" line, now it seems > that I get a syntax error: > > ---------- > Jan 15 14:14:28 mail cyrus/master[20778]: configuration file > /etc/cyrus.conf: bad character '_' in name on line 101 > Jan 15 14:14:28 mail cyrus/master[20778]: exiting > ---------- > > I think the configuration file already came with the "squatter_a" line > and I just removed the comment. Unless this is a bug in the > configuration file provided by Stretch for Cyrus 2.5.10: > > ---------- > 99 # reindex all mailboxes (fulltext) daily > 100 # DGB - 20190114 > 101 squatter_a cmd="/usr/sbin/cyrus squatter" at=0517 > ---------- Hmmmm... it seems this is a bug in the Debian Stretch configuration file because of what I see in the examples shown here [1]. The file also has the line "squatter_1" and it should be "squatter1". After that I managed to get the service is up and running (although I have still doubt about the "http" line). I have also come across this error now: ---------- Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message INIT to idled for mailbox user.jsmith: No such file or directory. Falling back to polling every 60 seconds. ---------- It seems this is preventing access to IMAP mailboxes. I am investigating the cause. Any contribution is welcome. Thanks in advance for your time. Kind regards, Daniel [1] https://www.cyrusimap.org/2.5/imap/admin/systemcommands/squatter.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From michael.menge at zdv.uni-tuebingen.de Tue Jan 15 15:47:09 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 15 Jan 2019 21:47:09 +0100 Subject: cyrus-imapd not starting after upgrade In-Reply-To: References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> Message-ID: <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> Quoting Daniel Bareiro : > On 15/1/19 15:05, Daniel Bareiro wrote: > >> The fact is that if I also comment out the "http" line, now it seems >> that I get a syntax error: >> >> ---------- >> Jan 15 14:14:28 mail cyrus/master[20778]: configuration file >> /etc/cyrus.conf: bad character '_' in name on line 101 >> Jan 15 14:14:28 mail cyrus/master[20778]: exiting >> ---------- >> >> I think the configuration file already came with the "squatter_a" line >> and I just removed the comment. Unless this is a bug in the >> configuration file provided by Stretch for Cyrus 2.5.10: >> >> ---------- >> 99 # reindex all mailboxes (fulltext) daily >> 100 # DGB - 20190114 >> 101 squatter_a cmd="/usr/sbin/cyrus squatter" at=0517 >> ---------- > > Hmmmm... it seems this is a bug in the Debian Stretch configuration file > because of what I see in the examples shown here [1]. The file also has > the line "squatter_1" and it should be "squatter1". > > After that I managed to get the service is up and running (although I > have still doubt about the "http" line). http is used for CalDAV / CardDav so is most likely in the cyrus-caldav package. This enables your cyrus server to store address books and calendars. > > I have also come across this error now: > > ---------- > Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message > INIT to idled for mailbox user.jsmith: No such file or directory. > Falling back to polling every 60 seconds. > ---------- > is idled (man idled) configured in cyrus.conf? > It seems this is preventing access to IMAP mailboxes. I am investigating > the cause. Any contribution is welcome. > It should not preventing access to IMAP mailboxes. This is a cpu friendly "inform me if there is new mail" feature which results in "Falling back to polling every 60 seconds." if it is missing > Thanks in advance for your time. > > > Kind regards, > Daniel > > [1] https://www.cyrusimap.org/2.5/imap/admin/systemcommands/squatter.html -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From daniel-listas at gmx.net Tue Jan 15 17:27:41 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Tue, 15 Jan 2019 19:27:41 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> Message-ID: Hi, Michael. Thanks for your reply. On 15/1/19 17:47, Michael Menge wrote: >> After that I managed to get the service is up and running (although I >> have still doubt about the "http" line). > http is used for CalDAV / CardDav so is most likely in the cyrus-caldav > package. > This enables your cyrus server to store address books and calendars. It seems that "http" does indeed enable CalDAV / CardDAV support with the cyrus-caldav package. I had not thought of it :-) After installing the package and uncommenting the "http" line, the error was not shown. I will need this to synchronize calendars and tasks. I guess it would have to work with this but Thunderbird shows a message saying "the calendar is momentarily unavailable". It's weird because it brought the information from Horde calendar and in fact I added a new event in Thunderbird and it was shown by Horde. But when I pressed the sync button this message was shown with an exclamation mark. But this has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm thinking if it has to do with something on the side of Thunderbird 60.x or Horde or the result of the interaction of both. >> I have also come across this error now: >> >> ---------- >> Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message >> INIT to idled for mailbox user.jsmith: No such file or directory. >> Falling back to polling every 60 seconds. >> ---------- > is idled (man idled) configured in cyrus.conf? When I upgraded to Stretch I initially installed the versions of configurations provided by the package maintainer and then passed the changes from the previous configuration files to the new versions of the configuration files. I thought that this would be the best. The "idled" line was commented out with Cyrus 2.4.17 and I don't remember having ever seen those messages in the log. So here I also kept it commented out (as it came initially). Maybe this is because something changed in the new version of Cyrus. Now I have activated that line and the message has vanished. >> It seems this is preventing access to IMAP mailboxes. I am investigating >> the cause. Any contribution is welcome. > It should not preventing access to IMAP mailboxes. This is a cpu friendly > "inform me if there is new mail" feature which results in > "Falling back to polling every 60 seconds." if it is missing Sorry. I expressed myself inaccurately. The content of the mailboxes is accessed. What I meant is it seems that from time to time the connection is lost. This is what I see in the log when that happens: ---------- Jan 15 19:12:22 mail cyrus/imaps[27754]: Fatal error: Exists wrong 257 256 11 16585 Jan 15 19:12:22 mail cyrus/master[26739]: process type:SERVICE name:imaps path:/usr/lib/cyrus/bin/imapd age:0.265s pid:27754 exited, status 75 ---------- For example, that happens when I try to access a folder with more than 2000 messages. I am investigating the cause. If you know what it can be, any info will be welcome. It's weird because I don't remember seeing this with Cyrus 2.4.17. Thanks for your time. Kind regards, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From michael.menge at zdv.uni-tuebingen.de Wed Jan 16 02:19:30 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 16 Jan 2019 08:19:30 +0100 Subject: cyrus-imapd not starting after upgrade In-Reply-To: References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> Message-ID: <20190116081930.Horde._6_XXBw8c7hNCIU3s__dRAk@webmail.uni-tuebingen.de> Quoting Daniel Bareiro : > Hi, Michael. > > Thanks for your reply. > > On 15/1/19 17:47, Michael Menge wrote: > >>> After that I managed to get the service is up and running (although I >>> have still doubt about the "http" line). > >> http is used for CalDAV / CardDav so is most likely in the cyrus-caldav >> package. >> This enables your cyrus server to store address books and calendars. > > It seems that "http" does indeed enable CalDAV / CardDAV support with > the cyrus-caldav package. I had not thought of it :-) After installing > the package and uncommenting the "http" line, the error was not shown. I > will need this to synchronize calendars and tasks. > > I guess it would have to work with this but Thunderbird shows a message > saying "the calendar is momentarily unavailable". It's weird because it > brought the information from Horde calendar and in fact I added a new > event in Thunderbird and it was shown by Horde. But when I pressed the > sync button this message was shown with an exclamation mark. But this > has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm > thinking if it has to do with something on the side of Thunderbird 60.x > or Horde or the result of the interaction of both. > Horde (Kronolith and Turba) have there own ClaDAV / CardDav Server, AFAIK Horde is at the moment not able to use cyrus as an Backend for its calendars and address books. Horde had support for Kolab, and the CalDAV / CardDAV in Cyrus is based on what Kolab did there. But the support in Horde has not been updated yet. So most likely you have configured your thunderbird to synchronize to horde and do not use the cyrus CalDAV / CardDAV feature >>> I have also come across this error now: >>> >>> ---------- >>> Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message >>> INIT to idled for mailbox user.jsmith: No such file or directory. >>> Falling back to polling every 60 seconds. >>> ---------- > >> is idled (man idled) configured in cyrus.conf? > > When I upgraded to Stretch I initially installed the versions of > configurations provided by the package maintainer and then passed the > changes from the previous configuration files to the new versions of the > configuration files. I thought that this would be the best. > > The "idled" line was commented out with Cyrus 2.4.17 and I don't > remember having ever seen those messages in the log. So here I also kept > it commented out (as it came initially). Maybe this is because something > changed in the new version of Cyrus. Now I have activated that line and > the message has vanished. > >>> It seems this is preventing access to IMAP mailboxes. I am investigating >>> the cause. Any contribution is welcome. > >> It should not preventing access to IMAP mailboxes. This is a cpu friendly >> "inform me if there is new mail" feature which results in >> "Falling back to polling every 60 seconds." if it is missing > > Sorry. I expressed myself inaccurately. The content of the mailboxes is > accessed. What I meant is it seems that from time to time the connection > is lost. This is what I see in the log when that happens: > > ---------- > Jan 15 19:12:22 mail cyrus/imaps[27754]: Fatal error: Exists wrong 257 > 256 11 16585 > Jan 15 19:12:22 mail cyrus/master[26739]: process type:SERVICE > name:imaps path:/usr/lib/cyrus/bin/imapd age:0.265s pid:27754 exited, > status 75 > ---------- > > For example, that happens when I try to access a folder with more than > 2000 messages. I am investigating the cause. If you know what it can be, > any info will be welcome. It's weird because I don't remember seeing > this with Cyrus 2.4.17. > > Thanks for your time. > > Kind regards, > Daniel -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From michael.menge at zdv.uni-tuebingen.de Wed Jan 16 06:12:47 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 16 Jan 2019 12:12:47 +0100 Subject: prblems rebuilding conversations db Message-ID: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> Hi, because conversations db seems to be required for search indexes, I enabled this option on our production servers today and tried to rebuild the conversations db for all users with ctl_conversationsdb -v -b UID For most users this did take less than a second. But for some users this process would not finish. I did kill one process after about 30 Minutes (most others after 3 Minutes). The UserID.conversations has grown over 2 GB (the mailbox itself has only ~700 MB of mails, and the conversations files from finnished rebuilds are less then 20 MB) Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock user.UserID for 1.5 seconds" every few seconds. I did try reconstruct -r -G user/UserID, followed by ctl_conversationsdb -v -z UID and tried to rebuild the db again. Reconstruct reported no errors, and the new rebuild hat the same problem. Has anybody seen a similar problem? Any ideas how to fix this? We are running cyrus-imapd 3.0.8 Kind regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From daniel-listas at gmx.net Wed Jan 16 09:01:57 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Wed, 16 Jan 2019 11:01:57 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: References: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net> Message-ID: <4fecab71-f107-1d0f-4e13-5d5b8be5ed0d@gmx.net> Hi, Andrew. Thanks for your reply. On 15/1/19 13:57, Andrew Morgan wrote: >> I'm not sure what the problem is but that "invalid magic header" makes >> me think that maybe it changed the header format of >> /var/lib/cyrus/tls_sessions.db and the migration process did not do the >> corresponding conversion. Can that be the reason why it doesn't start or >> I'm missing something else? Any ideas that can bring more light? >> >> The associated problem is that because of this it seems that Postfix can >> not deliver the mails since there is no /var/run/cyrus/socket/lmtp. > It wants tls_sessions.db to be a twoskip-format file, but the current > format is skiplist.? However, it was able to detect this and open it as > skiplist.? You can fix this issue by stopping Cyrus, removing > tls_sessions.db, and starting Cyrus. Thank you for this. It seems that Cyrus does not automatically create this file but after creating it manually, it starts without problems. > However, your real problem seems to be the missing nntp executable: > > Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for > service 'nntp' > > Do you use NNTP?? You could comment it out of cyrus.conf in order to get > the rest of Cyrus up and running. Thank you! Indeed that was the problem as you will have seen in a previous message. > Take a look at the release notes for v2.5.0: > > ? https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.0.html > > It covers important changes from v2.4 to v2.5.? You may need to update > your cyrus.conf and imapd.conf files. Thanks for the reference. I will read it to see the changes introduced in this new version. Now I am having this problem when trying to open a folder inside one of the mailboxes: ---------- Jan 16 10:47:17 mail cyrus/imaps[10794]: Fatal error: Exists wrong 257 256 22 16596 Jan 16 10:47:17 mail cyrus/master[2694]: process type:SERVICE name:imaps path:/usr/lib/cyrus/bin/imapd age:0.298s pid:10794 exited, status 75 ---------- I was not able to find the cause yet. Maybe a problem with repeated messages (that "257 256" makes me think about that) causes a process crash? This had never happened to me with 2.4.17. Apparently, it only happens with that folder. Yesterday I deleted all the messages there using Thunderbird (they were cron jobs mails) and now I would have to see the cron jobs that ran today, but I do not see anything. When I try to open the folder Thunderbird displays a message saying "Server [mailserver] has disconnected. The server may have gone down or there may be a network problem". Maybe the information in the folder is in an inconsistent state for some reason after the upgrade? Thanks for your time. Kind regards, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From Albert.Shih at obspm.fr Wed Jan 16 10:15:35 2019 From: Albert.Shih at obspm.fr (Albert Shih) Date: Wed, 16 Jan 2019 16:15:35 +0100 Subject: Big problem with replication Message-ID: <20190116151535.GC1667@io.chezmoi.fr> Hi everyone. I've got some big issue with replication. I've master --- replica ---> slave_1 --- replica ---> slave_2 The replication between master and slave_1 work nice. Between slave_1 and slave_2 I've got some issue (log to big after network failure and work nagios_supervision). So now I'm trying to build a new slave_3 to replace slave_2. And I'm unable to launch sync_client. Each time I try to manually launch I got [root at imap-mirror-p /bals/DELETED]# /usr/local/cyrus/sbin/sync_client -S slave_3 -A -v MAILBOXES DELETED.DIO.5AEAD6F9 Error from do_user(DELETED.DIO.5AEAD6F9): bailing out! [root at imap-mirror-p /bals/DELETED]# and the DIO folder don't event exist [root at imap-mirror-p /bals/DELETED]# ls DIO ls: DIO: No such file or directory [root at imap-mirror-p /bals/DELETED]# Any help would be *very* welcome ;-) Regards. -- Albert SHIH DIO b?timent 15 Observatoire de Paris Heure local/Local time: Wed Jan 16 16:08:47 CET 2019 From egoitz at sarenet.es Wed Jan 16 11:10:30 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Wed, 16 Jan 2019 17:10:30 +0100 Subject: Big problem with replication In-Reply-To: <20190116151535.GC1667@io.chezmoi.fr> References: <20190116151535.GC1667@io.chezmoi.fr> Message-ID: <1ceda161b5e3a78ed876e2a1da010fdb@sarenet.es> Good afternoon, I would try doing it user by user (with -u). This way you would have all synced except the problematic mailbox. Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 16-01-2019 16:15, Albert Shih escribi?: > Hi everyone. > > I've got some big issue with replication. > > I've > > master --- replica ---> slave_1 --- replica ---> slave_2 > > The replication between master and slave_1 work nice. > > Between slave_1 and slave_2 I've got some issue (log to big after network > failure and work nagios_supervision). > > So now I'm trying to build a new slave_3 to replace slave_2. And I'm unable > to launch sync_client. Each time I try to manually launch I got > > [root at imap-mirror-p /bals/DELETED]# /usr/local/cyrus/sbin/sync_client -S slave_3 -A -v > MAILBOXES DELETED.DIO.5AEAD6F9 > Error from do_user(DELETED.DIO.5AEAD6F9): bailing out! > [root at imap-mirror-p /bals/DELETED]# > > and the DIO folder don't event exist > > [root at imap-mirror-p /bals/DELETED]# ls DIO > ls: DIO: No such file or directory > [root at imap-mirror-p /bals/DELETED]# > > Any help would be *very* welcome ;-) > > Regards. > > -- > Albert SHIH > DIO b?timent 15 > Observatoire de Paris > Heure local/Local time: > Wed Jan 16 16:08:47 CET 2019 > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Albert.Shih at obspm.fr Wed Jan 16 12:26:11 2019 From: Albert.Shih at obspm.fr (Albert Shih) Date: Wed, 16 Jan 2019 18:26:11 +0100 Subject: Big problem with replication In-Reply-To: <1ceda161b5e3a78ed876e2a1da010fdb@sarenet.es> References: <20190116151535.GC1667@io.chezmoi.fr> <1ceda161b5e3a78ed876e2a1da010fdb@sarenet.es> Message-ID: <20190116172611.GA2582@io.chezmoi.fr> Le 16/01/2019 ? 17:10:30+0100, Egoitz Aurrekoetxea a ?crit > Good afternoon, > > > I would try doing it user by user (with -u). This way you would have all synced > except the problematic mailbox. Hi, thanks for the help. I got some progress in my problem : > [root at imap-mirror-p /bals/DELETED]# /usr/local/cyrus/sbin/sync_client -S > slave_3 -A -v > MAILBOXES DELETED.DIO.5AEAD6F9 > Error from do_user(DELETED.DIO.5AEAD6F9): bailing out! > [root at imap-mirror-p /bals/DELETED]# > > and the DIO folder don't event exist > > [root at imap-mirror-p /bals/DELETED]# ls DIO > ls: DIO: No such file or directory > [root at imap-mirror-p /bals/DELETED]# > For some strange reason the I was unable to destroy the mailbox either (with cyradm), so I copy some junk mailbox on the filesystem, and run reconstruct and finally I'm was able to destroy those mailbox. But that's not really solve my problem because now when I run the sync_client he crash at the beginning with a shared mailbox. It stop with [root at imap-mirror-p /usr/home/jas-adm]# /usr/local/cyrus/sbin/sync_client -S imap-mirror-m-tmp -A -v MAILBOXES shared.***** MAILBOX shared.***** Error from do_user(shared.*****): bailing out! [root at imap-mirror-p /usr/home/jas-adm]# I've no idea if it's normal or not. I don't think so, because the first level (Master -- replica --> slave_1) work well event with thoses shared_mailbox. Any help would be very welcome. Regards. -- Albert SHIH Heure local/Local time: Wed Jan 16 18:11:53 CET 2019 From daniel-listas at gmx.net Wed Jan 16 20:24:23 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Wed, 16 Jan 2019 22:24:23 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <20190116081930.Horde._6_XXBw8c7hNCIU3s__dRAk@webmail.uni-tuebingen.de> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> <20190116081930.Horde._6_XXBw8c7hNCIU3s__dRAk@webmail.uni-tuebingen.de> Message-ID: <36909942-24e4-75f1-7494-8c7741f0990f@gmx.net> Hi, Michael. Thanks for your reply. On 16/1/19 04:19, Michael Menge wrote: >>>> After that I managed to get the service is up and running (although I >>>> have still doubt about the "http" line). >>> http is used for CalDAV / CardDav so is most likely in the cyrus-caldav >>> package. >>> This enables your cyrus server to store address books and calendars. >> It seems that "http" does indeed enable CalDAV / CardDAV support with >> the cyrus-caldav package. I had not thought of it :-) After installing >> the package and uncommenting the "http" line, the error was not shown. I >> will need this to synchronize calendars and tasks. >> >> I guess it would have to work with this but Thunderbird shows a message >> saying "the calendar is momentarily unavailable". It's weird because it >> brought the information from Horde calendar and in fact I added a new >> event in Thunderbird and it was shown by Horde. But when I pressed the >> sync button this message was shown with an exclamation mark. But this >> has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm >> thinking if it has to do with something on the side of Thunderbird 60.x >> or Horde or the result of the interaction of both. > Horde (Kronolith and Turba) have there own ClaDAV / CardDav Server, > AFAIK Horde is at the moment not able to use cyrus as an Backend for > its calendars and address books. > > Horde had support for Kolab, and the CalDAV / CardDAV in Cyrus is > based on what Kolab did there. But the support in Horde has not been > updated yet. > > So most likely you have configured your thunderbird to synchronize > to horde and do not use the cyrus CalDAV / CardDAV feature Yes, I did it using the HTTP address provided by Horde for the corresponding calendar. I thought Thunderbird needed that Cyrus support from CalDAV / CarDAV to interact with Horde. I think I would just be missing solve the crash of the process when trying to access a specific folder (I think it just happens with that). ---------- Jan 16 10:47:17 mail cyrus/imaps[10794]: Fatal error: Exists wrong 257 256 22 16596 Jan 16 10:47:17 mail cyrus/master[2694]: process type:SERVICE name:imaps path:/usr/lib/cyrus/bin/imapd age:0.298s pid:10794 exited, status 75 ---------- I was not able to find the cause yet. Maybe a problem with repeated messages (that "257 256" makes me think about that) causes a process crash? This had never happened to me with 2.4.17. Apparently, it only happens with that folder. Yesterday I deleted all the messages there using Thunderbird (they were cron jobs mails) and now I would have to see the cron jobs that ran today, but I don't see anything. When I try to open the folder, Thunderbird displays a message saying "Server [mailserver] has disconnected. The server may have gone down or there may be a network problem". Maybe the information in the folder is in an inconsistent state for some reason after the upgrade? Thanks for your time. Kind regards, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From jan at horde.org Thu Jan 17 03:32:37 2019 From: jan at horde.org (Jan Schneider) Date: Thu, 17 Jan 2019 08:32:37 +0000 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <36909942-24e4-75f1-7494-8c7741f0990f@gmx.net> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> <20190116081930.Horde._6_XXBw8c7hNCIU3s__dRAk@webmail.uni-tuebingen.de> <36909942-24e4-75f1-7494-8c7741f0990f@gmx.net> Message-ID: <20190117083237.Horde.4KPDfmkMzAmO3FQJU1xBi2R@yunosh.horde.org> Zitat von Daniel Bareiro : > Hi, Michael. Thanks for your reply. > > On 16/1/19 04:19, Michael Menge wrote: > >>>>> After that I managed to get the service is up and running (although I >>>>> have still doubt about the "http" line). > >>>> http is used for CalDAV / CardDav so is most likely in the cyrus-caldav >>>> package. >>>> This enables your cyrus server to store address books and calendars. > >>> It seems that "http" does indeed enable CalDAV / CardDAV support with >>> the cyrus-caldav package. I had not thought of it :-) After installing >>> the package and uncommenting the "http" line, the error was not shown. I >>> will need this to synchronize calendars and tasks. >>> >>> I guess it would have to work with this but Thunderbird shows a message >>> saying "the calendar is momentarily unavailable". It's weird because it >>> brought the information from Horde calendar and in fact I added a new >>> event in Thunderbird and it was shown by Horde. But when I pressed the >>> sync button this message was shown with an exclamation mark. But this >>> has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm >>> thinking if it has to do with something on the side of Thunderbird 60.x >>> or Horde or the result of the interaction of both. > >> Horde (Kronolith and Turba) have there own ClaDAV / CardDav Server, >> AFAIK Horde is at the moment not able to use cyrus as an Backend for >> its calendars and address books. >> >> Horde had support for Kolab, and the CalDAV / CardDAV in Cyrus is >> based on what Kolab did there. But the support in Horde has not been >> updated yet. >> >> So most likely you have configured your thunderbird to synchronize >> to horde and do not use the cyrus CalDAV / CardDAV feature > > Yes, I did it using the HTTP address provided by Horde for the > corresponding calendar. I thought Thunderbird needed that Cyrus support > from CalDAV / CarDAV to interact with Horde. > > I think I would just be missing solve the crash of the process when > trying to access a specific folder (I think it just happens with that). > > ---------- > Jan 16 10:47:17 mail cyrus/imaps[10794]: Fatal error: Exists wrong 257 > 256 22 16596 > Jan 16 10:47:17 mail cyrus/master[2694]: process type:SERVICE name:imaps > path:/usr/lib/cyrus/bin/imapd age:0.298s pid:10794 exited, status 75 > ---------- > > I was not able to find the cause yet. Maybe a problem with repeated > messages (that "257 256" makes me think about that) causes a process > crash? This had never happened to me with 2.4.17. Apparently, it only > happens with that folder. Yesterday I deleted all the messages there > using Thunderbird (they were cron jobs mails) and now I would have to > see the cron jobs that ran today, but I don't see anything. When I try > to open the folder, Thunderbird displays a message saying "Server > [mailserver] has disconnected. The server may have gone down or there > may be a network problem". Maybe the information in the folder is in an > inconsistent state for some reason after the upgrade? > > Thanks for your time. > > > Kind regards, > Daniel Just for clarification: you can use Cyrus as a backend for address books and calendars in Horde, though not via CalDAV/CardDAV but via its generic IMAP backend (which has evolved from the Kolab backends). There is also a generic CalDAV backend support in Kronolith, but I haven't tested this with Cyrus' CalDAV server yet. -- Jan Schneider The Horde Project https://www.horde.org/ From daniel-listas at gmx.net Thu Jan 17 10:30:05 2019 From: daniel-listas at gmx.net (Daniel Bareiro) Date: Thu, 17 Jan 2019 12:30:05 -0300 Subject: cyrus-imapd not starting after upgrade In-Reply-To: <20190117083237.Horde.4KPDfmkMzAmO3FQJU1xBi2R@yunosh.horde.org> References: <20190115175415.Horde.-GLtt7JbsfnaXgTZr8YWpka@webmail.uni-tuebingen.de> <3b583305-cb6f-42bf-238e-19c8b9f38c25@gmx.net> <20190115214709.Horde.dDWNCKIEjEimc1Dh4gueQOz@webmail.uni-tuebingen.de> <20190116081930.Horde._6_XXBw8c7hNCIU3s__dRAk@webmail.uni-tuebingen.de> <36909942-24e4-75f1-7494-8c7741f0990f@gmx.net> <20190117083237.Horde.4KPDfmkMzAmO3FQJU1xBi2R@yunosh.horde.org> Message-ID: <44499e4e-fb6c-db1b-5b90-ce96f4e02568@gmx.net> Hi, Jan. Good to see you here. Thanks for your reply. On 17/1/19 05:32, Jan Schneider wrote: >>>>>> After that I managed to get the service is up and running (although I >>>>>> have still doubt about the "http" line). >>>>> http is used for CalDAV / CardDav so is most likely in the >>>>> cyrus-caldav >>>>> package. >>>>> This enables your cyrus server to store address books and calendars. >>>> It seems that "http" does indeed enable CalDAV / CardDAV support with >>>> the cyrus-caldav package. I had not thought of it :-) After installing >>>> the package and uncommenting the "http" line, the error was not >>>> shown. I >>>> will need this to synchronize calendars and tasks. >>>> >>>> I guess it would have to work with this but Thunderbird shows a message >>>> saying "the calendar is momentarily unavailable". It's weird because it >>>> brought the information from Horde calendar and in fact I added a new >>>> event in Thunderbird and it was shown by Horde. But when I pressed the >>>> sync button this message was shown with an exclamation mark. But this >>>> has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm >>>> thinking if it has to do with something on the side of Thunderbird 60.x >>>> or Horde or the result of the interaction of both. >>> Horde (Kronolith and Turba) have there own ClaDAV / CardDav Server, >>> AFAIK Horde is at the moment not able to use cyrus as an Backend for >>> its calendars and address books. >>> >>> Horde had support for Kolab, and the CalDAV / CardDAV in Cyrus is >>> based on what Kolab did there. But the support in Horde has not been >>> updated yet. >>> >>> So most likely you have configured your thunderbird to synchronize >>> to horde and do not use the cyrus CalDAV / CardDAV feature >> >> Yes, I did it using the HTTP address provided by Horde for the >> corresponding calendar. I thought Thunderbird needed that Cyrus support >> from CalDAV / CarDAV to interact with Horde. > Just for clarification: you can use Cyrus as a backend for address books > and calendars in Horde, though not via CalDAV/CardDAV but via its > generic IMAP backend (which has evolved from the Kolab backends). There > is also a generic CalDAV backend support in Kronolith, but I haven't > tested this with Cyrus' CalDAV server yet. Thanks for the observation. I think that explains why when I have a notification window for a calendar event in Thunderbird and I press a button in that Window, the Postfix log shows an IMAP access. Something like this: ---------- Jan 17 12:03:06 mail cyrus/imap[3326]: starttls: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits new) no authentication Jan 17 12:03:06 mail cyrus/imap[3326]: login: localhost [::1] dbareiro PLAIN+TLS User logged in SESSIONID= ---------- By the way, if you have a moment to check the last message I sent in the thread "Invalid VObject, line 1 did not follow the icalendar / vcard format" in the Horde list, I would appreciate that. It's my response to an email from Michael Rubinsky. In case it's helpful, I'm using Horde Groupware Webmail Edition 5.2.22. Thanks for your time. Kind regards, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From Albert.Shih at obspm.fr Fri Jan 18 09:26:58 2019 From: Albert.Shih at obspm.fr (Albert Shih) Date: Fri, 18 Jan 2019 15:26:58 +0100 Subject: Sync crash with bailing out Message-ID: <20190118142658.GB15076@io.chezmoi.fr> Hi everyone, My sync_client crash just after start with this message [root at imap-mirror-p /var/imap/sync/log]# /usr/local/cyrus/sbin/sync_client -S imap-mirror-m-tmp -A -v MAILBOXES shared.some_folder MAILBOX shared.some_folder Error from do_user(shared.some_folder): bailing out! [root at imap-mirror-p /var/imap/sync/log]# And indeed I don't have a user name shared.some_folder.... Any help..... ? Regards -- Albert SHIH Observatoire de Paris xmpp: jas at obspm.fr Heure local/Local time: Fri Jan 18 15:21:12 CET 2019 From miguel at prodemge.gov.br Fri Jan 18 16:20:49 2019 From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira) Date: Fri, 18 Jan 2019 19:20:49 -0200 Subject: Mailboxes with slow access Message-ID: <6f8e-5c424300-5-dbb7a80@155956654> Hi folks! I've had a problem with some huge mailboxes, with a lot of folders and subfolders. Problably thousand of folders. The access on these mailboxes is too slow, if you change from INBOX to another folder called CYRUS, even if CYRUS folder has just 4 messages for example the access is slow. I was trying to find out? anything on internet about this issue and I read that a parameter maxword on imapd.conf? could be the solution. Nowadays in our cyrus-imap murder all backend servers and frontend servers have the maxword parameter set to default value which is 128kb. Based on that information, I'd like to know if I increase this parameter to a high value like 1Mb for instance it could fix the issue about slow access and which considerations about memory, disk and cpu resources I must take care if I set up this value in my environment. Another question is, this value must be setting in all backends and frontends servers? Thanks in advance Regards -- Miguel Moreira Gerente DPR/SRE/GSR - Ger?ncia de Servi?os de Rede +55(31)3339-1401 PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrl at psfc.mit.edu Fri Jan 18 16:26:01 2019 From: mrl at psfc.mit.edu (Mark London) Date: Fri, 18 Jan 2019 16:26:01 -0500 Subject: Is there any to reconstruct a mailbox, if used just have the email files, but none of the original cyrus.* files? Message-ID: Hi - Is there any to reconstruct a mailbox, if you just have the email files, but none of the original cyrus.* files? Don't ask me why, it's a very long story. As an aside, has anyone ever had a situation where emails were los,t when someone used Apple Mail to transfer a lot of files from one folder to another? We had this happen. Someone tried to move a years worth of emails, 22K or so, to a new folder. Only a few files emails ended up in the new folders, and the rest were totally deleted. I confirmed via my backup logs, that this was the case. The same user tried the same operation the year before, and the same thing happened. i tried using Thunderbird, and had no problems. Just curious. Thanks. Mark London mrl at psfc.mit.edu From boutilpj at ednet.ns.ca Fri Jan 18 19:50:26 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 18 Jan 2019 20:50:26 -0400 Subject: Is there any to reconstruct a mailbox, if used just have the email files, but none of the original cyrus.* files? In-Reply-To: References: Message-ID: <494e11f3-9190-894e-cc81-ccc822238e98@ednet.ns.ca> On 1/18/19 5:26 PM, Mark London wrote: > Hi - Is there any to reconstruct a mailbox, if you just have the email > files, but none of the original cyrus.* files??? Don't ask me why, it's > a very long story. I believe you can just recreate the mailbox (using cyradm or similar) and then run the reconstruct. > > As an aside, has anyone ever had a situation where emails were los,t > when someone used Apple Mail to transfer a lot of files from one folder > to another??? We had this happen.?? Someone tried to move a years worth > of emails,? 22K or so, to a new folder.?? Only a few files emails ended > up in the new folders, and the rest were totally deleted.? I confirmed > via my backup logs, that this was the case. The same user tried the same > operation the year before, and the same thing happened.?? i tried using > Thunderbird, and had no problems. Just curious.? Thanks. > > Mark London > mrl at psfc.mit.edu > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From ml at netfence.it Sat Jan 19 06:14:56 2019 From: ml at netfence.it (Andrea Venturoli) Date: Sat, 19 Jan 2019 12:14:56 +0100 Subject: Sieve script not working Message-ID: Hello. I'm using sieve scripts (among other things) to move messages that MIMEDefang marked as spam into a dedicated folder. My scripts look like this: > require ["fileinto"]; > require ["regex"]; > > if anyof (header :regex "X-Spam-Score" "^[1-9][0-9]\.", > header :contains "X-Attachment-Dropped" "") > {fileinto "INBOX.Spam.Received";} Thus a message with the following header should go directly into the "Spam/Received" folder: > X-Spam-Score: 26.012 (**************************) BITCOIN_SPAM_07,DCC_CHECK,DOS_OE_TO_MX,FSL_BULK_SIG,HDR_ORDER_FTSDMCXX_DIRECT,HDR_ORDER_FTSDMCXX_NORDNS,HELO_DYNAMIC_HCC,HELO_DYNAMIC_IPADDR2,HTML_MESSAGE,MIMEOLE_DIRECT_TO_MX,NO_FM_NAME_IP_HOSTN,RATWARE_NO_RDNS,RDNS_NONE,TO_EQ_FM_DIRECT_MX,ZMIde_OutlookExpress I'm doing this for three mailboxes: it works on two of them, not on the third one. I checked the script is there and the folder exists; now I'm out of ideas on what to try. Is there some way to get a sieve log or to debug it? bye & Thanks av. From nic at onlight.com Sat Jan 19 18:13:44 2019 From: nic at onlight.com (Nic Bernstein) Date: Sat, 19 Jan 2019 17:13:44 -0600 Subject: Sieve script not working In-Reply-To: References: Message-ID: <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com> Andrea, It would help to know which version of Cyrus you're having difficulty with, and what setting you're using for 'unixhierarchysep'?? If you've recently upgraded to 3.X, for example, have you followed the instructions and converted your sieve scripts? Cheers, ??? -nic On 1/19/19 5:14 AM, Andrea Venturoli wrote: > Hello. > > I'm using sieve scripts (among other things) to move messages that > MIMEDefang marked as spam into a dedicated folder. > > My scripts look like this: > >> require ["fileinto"]; >> require ["regex"]; >> >> if anyof (header :regex "X-Spam-Score" "^[1-9][0-9]\.", >> ????????? header :contains "X-Attachment-Dropped" "") >> ? {fileinto "INBOX.Spam.Received";} > > Thus a message with the following header should go directly into the > "Spam/Received" folder: >> X-Spam-Score: 26.012 (**************************) >> BITCOIN_SPAM_07,DCC_CHECK,DOS_OE_TO_MX,FSL_BULK_SIG,HDR_ORDER_FTSDMCXX_DIRECT,HDR_ORDER_FTSDMCXX_NORDNS,HELO_DYNAMIC_HCC,HELO_DYNAMIC_IPADDR2,HTML_MESSAGE,MIMEOLE_DIRECT_TO_MX,NO_FM_NAME_IP_HOSTN,RATWARE_NO_RDNS,RDNS_NONE,TO_EQ_FM_DIRECT_MX,ZMIde_OutlookExpress > > > > I'm doing this for three mailboxes: it works on two of them, not on > the third one. > I checked the script is there and the folder exists; now I'm out of > ideas on what to try. > > Is there some way to get a sieve log or to debug it? > > ?bye & Thanks > ????av. > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Nic Bernstein nic at onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 From ml at netfence.it Sun Jan 20 09:21:38 2019 From: ml at netfence.it (Andrea Venturoli) Date: Sun, 20 Jan 2019 15:21:38 +0100 Subject: Sieve script not working In-Reply-To: <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com> References: <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com> Message-ID: On 1/20/19 12:13 AM, Nic Bernstein wrote: > Andrea, > It would help to know which version of Cyrus you're having difficulty > with, and what setting you're using for 'unixhierarchysep'? Sorry, I obviously should have said it from the start: I'm using version 2.5.12 on FreeBSD 11.2/amd64. I'm not using 'unixhierarchysep', so the "default" separator shoud be a dot. I'm also not using "altnamespace". bye & Thanks av. From ellie at fastmail.com Sun Jan 20 20:33:40 2019 From: ellie at fastmail.com (ellie timoney) Date: Mon, 21 Jan 2019 12:33:40 +1100 Subject: Mailboxes with slow access In-Reply-To: <6f8e-5c424300-5-dbb7a80@155956654> References: <6f8e-5c424300-5-dbb7a80@155956654> Message-ID: <1548034420.3823238.1639560584.33DA656D@webmail.messagingengine.com> Hi Miguel, On Sat, Jan 19, 2019, at 8:20 AM, Miguel Mucio Santos Moreira wrote: > Hi folks! > > I've had a problem with some huge mailboxes, with a lot of folders and > subfolders. Problably thousand of folders.> The access on these mailboxes is too slow, if you change from INBOX to > another folder called CYRUS, even if CYRUS folder has just 4 messages > for example the access is slow. This sounds like you don't have enough RAM to hold the metadata for these mailboxes in memory, and so the system is swapping while accessing them. You could find the imapd process that is servicing this connection and look at its virtual memory size, and compare that to the amount of physical RAM in your system, to see whether this is the case. Or, if you post up the size of these mailboxes' cyrus.header, cyrus.index, cyrus.annotations, cyrus.cache (...) files and the size of their [user].conversations file (if you have conversations enabled), then maybe someone will be able to suggest a RAM size that would support this. You mention being in a murder environment -- I'm not sure if you will need to increase the RAM on just the backend for these mailboxes, or the frontends as well. Maybe someone running a large murder can chime in with how they provision their servers? Sorry I don't have specific numbers for you, but hopefully this points you in the right direction! > I was trying to find out anything on internet about this issue and I > read that a parameter maxword on imapd.conf could be the solution.> Nowadays in our cyrus-imap murder all backend servers and > frontend servers have the maxword parameter set to default value > which is 128kb.> Based on that information, I'd like to know if I increase this > parameter to a high value like 1Mb for instance it could fix the issue > about slow access and which considerations about memory, disk and cpu > resources I must take care if I set up this value in my environment.> Another question is, this value must be setting in all backends and > frontends servers? imapd.conf(5) says: > maxword: 131072 > Maximum size of a single word for the parser. Default > 128k This setting won't have any effect on performance at all. It's just an upper limit for the IMAP parser -- if some IMAP client tries to send a single word that is longer than this, the imapd process serving their connection will exit immediately with a "word too long" error rather than trying to keep parsing. Normal traffic won't hit this limit, it's just protecting itself from (possibly malicious) garbage. The default should be never need changing really. Hope this helps, ellie -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Sun Jan 20 21:20:19 2019 From: ellie at fastmail.com (ellie timoney) Date: Mon, 21 Jan 2019 13:20:19 +1100 Subject: prblems rebuilding conversations db In-Reply-To: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> References: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> Message-ID: <1548037219.3832722.1639587944.75697E74@webmail.messagingengine.com> Hi Michael, On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote: > Hi, > > because conversations db seems to be required for search indexes, I > enabled this option > on our production servers today and tried to rebuild the conversations > db for all users with > > ctl_conversationsdb -v -b UID > > For most users this did take less than a second. But for some users > this process would > not finish. I did kill one process after about 30 Minutes (most others > after 3 Minutes). > The UserID.conversations has grown over 2 GB (the mailbox itself has > only ~700 MB of mails, > and the conversations files from finnished rebuilds are less then 20 MB) So, a 2GB-and-growing conversations db for a 700MB mailbox is a ratio of ~3:1, which seems obscene... For comparison, the Conversations.append_reply_200 cassandane test (two messages with 100 replies each) produces an 87Kb conversations db for a 1016Kb mailbox (ratio ~1:12, n.b inverted!), which makes your ~3:1 and growing ratio seem even more obscene... Obviously you can't share the bad conversations db because conversations db's are full of personally-identifying information. But I wonder if looking through it with `cyr_dbtool ... show` turns up any unusual patterns, especially in comparison to one of the good users? I wonder if it's somehow repeating itself? Bron, does this sound like anything you've seen before, that maybe got fixed on master but not backported to 3.0? > Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock > user.UserID for 1.5 seconds" > every few seconds. This message is logged when the lock is released, if that lock had been held for >1s. It's reporting that it held the lock for longer than it would like to have, but at least you know the lock has been released. I think this is telling you that user-visible performance on the mailbox would have been impacted while this was occurring (because imapd/lmtpd wouldn't be able to access their mailbox while ctl_conversationsdb was holding those locks), but also tells you that ctl_conversationsdb was doing the right thing, and not just holding a single lock for the entire job. Though, that raises the question: was the user accessing the mailbox while the rebuild was in progress? I wonder if their client is doing something funny and tripping things up? Cheers, ellie > I did try reconstruct -r -G user/UserID, followed by > ctl_conversationsdb -v -z UID > and tried to rebuild the db again. Reconstruct reported no errors, and > the new rebuild > hat the same problem. > > Has anybody seen a similar problem? > > Any ideas how to fix this? > > > We are running cyrus-imapd 3.0.8 > > > Kind regards > > Michael Menge > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen From falon at ruparpiemonte.it Mon Jan 21 05:49:11 2019 From: falon at ruparpiemonte.it (Marco) Date: Mon, 21 Jan 2019 11:49:11 +0100 Subject: squatter issue on Cyrus 2.4 Message-ID: <6d3deb22-1021-4e0c-06ce-6b47867ea979@ruparpiemonte.it> Hello, I have Cyrus 2.4 with thousand mailboxes and separated partitions for data and metadata, with squat files in metamaildata partitions. Every night at 2 I run "squatter -s -a -i" to update the indexes. Two nights ago a metamaildata partition filled out. Starting from 2am many GiB busy in few hours. I can't understand why. No errors in Cyrus logs. I didn't find the files which caused the busy space. Last night squatter didn't reproduce the error. Do you have ever experienced problems like this? Thank you very much for every hints. Kind Regards Marco ****************************************************** name : Cyrus IMAPD version : v2.4.17-Invoca-RPM-2.4.17-6.el6 d1df8aff 2012-12-01 vendor : Project Cyrus support-url: http://www.cyrusimap.org os : Linux os-version : 2.6.32-279.el6.x86_64 environment: Built w/Cyrus SASL 2.1.23 Running w/Cyrus SASL 2.1.23 Built w/OpenSSL 1.0.0-fips 29 Mar 2010 Running w/OpenSSL 1.0.0-fips 29 Mar 2010 Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.4 TCP Wrappers NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = idled ****************************************************** EVENTS { # Squatter squatter cmd="squatter -s -a -i" at=0200 ****************************************************** metapartition_files: header index cache expunge squat ****************************************************** From michael.menge at zdv.uni-tuebingen.de Mon Jan 21 08:32:07 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Mon, 21 Jan 2019 14:32:07 +0100 Subject: prblems rebuilding conversations db In-Reply-To: <1548037219.3832722.1639587944.75697E74@webmail.messagingengine.com> References: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> <1548037219.3832722.1639587944.75697E74@webmail.messagingengine.com> Message-ID: <20190121143207.Horde.oR1nN76B2p9NRilOsUDDwg4@webmail.uni-tuebingen.de> Hi Ellie Quoting ellie timoney : > Hi Michael, > > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote: >> Hi, >> >> because conversations db seems to be required for search indexes, I >> enabled this option >> on our production servers today and tried to rebuild the conversations >> db for all users with >> >> ctl_conversationsdb -v -b UID >> >> For most users this did take less than a second. But for some users >> this process would >> not finish. I did kill one process after about 30 Minutes (most others >> after 3 Minutes). >> The UserID.conversations has grown over 2 GB (the mailbox itself has >> only ~700 MB of mails, >> and the conversations files from finnished rebuilds are less then 20 MB) > > So, a 2GB-and-growing conversations db for a 700MB mailbox is a > ratio of ~3:1, which seems obscene... > thanks for confirming that this is not expected. > For comparison, the Conversations.append_reply_200 cassandane test > (two messages with 100 replies each) produces an 87Kb conversations > db for a 1016Kb mailbox (ratio ~1:12, n.b inverted!), which makes > your ~3:1 and growing ratio seem even more obscene... > > Obviously you can't share the bad conversations db because > conversations db's are full of personally-identifying information. > But I wonder if looking through it with `cyr_dbtool ... show` turns > up any unusual patterns, especially in comparison to one of the good > users? I wonder if it's somehow repeating itself? I was unable to run "cyr_dbtool ... show" or "ctl_conversationsdb -d" while ctl_conversationsdb -v -b was still running (waiting for a lock on conversations db file) After aborting "ctl_conversationsdb -b" "cyr_dbtool show" didn't show any duplicates. But the string command showed more occurrences than cyr_dbtool did (i used an hash value (d8087b76083ec5e7) I found in the output of cyr_dbtool # cyr_dbtool /path-to/userid.conversations skiplist show | grep d8087b76083ec5e7 | wc -l 3 # cyr_dbtool /path-to/userid.conversations skiplist show | grep Bd8087b76083ec5e7 | wc -l 1 # strings /path-to/userid.conversations | grep d8087b76083ec5e7 | wc -l 183 # strings /path-to/userid.conversations | grep Bd8087b76083ec5e7 | wc -l 91 ctl_conversationsdb -d did a cleanup because I did abort the "ctl_conversationsdb -b" process skiplist recovery /path-to/userid.conversations: found partial txn, not replaying skiplist: recovered /path-to/userid.conversations (0 records, 144 bytes) in 0 seconds skiplist: checkpointed /path-to/userid.conversations (0 records, 144 bytes) in 0.010 sec I did run ctl_conversationsd -b with strace and discovered that the inbox (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache cyrus.O_RDWR ) of that user was opened over and over again, no other folder for that user was accessed. For other users the inbox was only opened once and then the other folders followed. So not stopping "ctl_conversationsd -b", it would have run till my filesystem was full. > > Bron, does this sound like anything you've seen before, that maybe > got fixed on master but not backported to 3.0? > >> Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock >> user.UserID for 1.5 seconds" >> every few seconds. > > This message is logged when the lock is released, if that lock had > been held for >1s. It's reporting that it held the lock for longer > than it would like to have, but at least you know the lock has been > released. I think this is telling you that user-visible performance > on the mailbox would have been impacted while this was occurring > (because imapd/lmtpd wouldn't be able to access their mailbox while > ctl_conversationsdb was holding those locks), but also tells you > that ctl_conversationsdb was doing the right thing, and not just > holding a single lock for the entire job. > I am not sure if any other processes had a chance to acquire a lock. > Though, that raises the question: was the user accessing the mailbox > while the rebuild was in progress? I wonder if their client is > doing something funny and tripping things up? We did this while the server was accessible (didn't want to have several hour downtime for ~44000 users) But as I did my recent testing on a test-server without user access, only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during the conversation rebuild mentioned in this mail. So I don't think that user access is part of the problem. So for me it looks like we have two problems: 1. multiple entries for the same key in skiplist db files. As skiplist is a Key/Value store this should not be possible to have duplicate keys? Is twoskip a better alternative? 2. Why didn't ctl_conversationsdb continue to process the next mailbox but re-did the INBOX / the last mailbox I will try to run a debugger on ctl_conversationsdb and will test with twoskip as conversations db format I did notice one other problem, recreating the conversations db on the replica confused the sync protocol as the rebuild did increase the modseq. Many logs like "record mismatch with replica: %s more recent on replica" on the sync client and "higher modseq on replica" on the sync server is this intended? Regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From ellie at fastmail.com Mon Jan 21 17:16:04 2019 From: ellie at fastmail.com (ellie timoney) Date: Tue, 22 Jan 2019 09:16:04 +1100 Subject: prblems rebuilding conversations db In-Reply-To: <20190121143207.Horde.oR1nN76B2p9NRilOsUDDwg4@webmail.uni-tuebingen.de> References: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> <1548037219.3832722.1639587944.75697E74@webmail.messagingengine.com> <20190121143207.Horde.oR1nN76B2p9NRilOsUDDwg4@webmail.uni-tuebingen.de> Message-ID: <1548108964.1676215.1640323480.73A4EE8F@webmail.messagingengine.com> Hi Michael, On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote: > Hi Ellie > > Quoting ellie timoney : > > > Hi Michael, > > > > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote: > >> Hi, > >> > >> because conversations db seems to be required for search indexes, I > >> enabled this option > >> on our production servers today and tried to rebuild the conversations > >> db for all users with > >> > >> ctl_conversationsdb -v -b UID > >> > >> For most users this did take less than a second. But for some users > >> this process would > >> not finish. I did kill one process after about 30 Minutes (most others > >> after 3 Minutes). > >> The UserID.conversations has grown over 2 GB (the mailbox itself has > >> only ~700 MB of mails, > >> and the conversations files from finnished rebuilds are less then 20 MB) > > > > So, a 2GB-and-growing conversations db for a 700MB mailbox is a > > ratio of ~3:1, which seems obscene... > > > > thanks for confirming that this is not expected. > > > For comparison, the Conversations.append_reply_200 cassandane test > > (two messages with 100 replies each) produces an 87Kb conversations > > db for a 1016Kb mailbox (ratio ~1:12, n.b inverted!), which makes > > your ~3:1 and growing ratio seem even more obscene... > > > > Obviously you can't share the bad conversations db because > > conversations db's are full of personally-identifying information. > > But I wonder if looking through it with `cyr_dbtool ... show` turns > > up any unusual patterns, especially in comparison to one of the good > > users? I wonder if it's somehow repeating itself? > > I was unable to run "cyr_dbtool ... show" or "ctl_conversationsdb -d" > while ctl_conversationsdb -v -b was still running > (waiting for a lock on conversations db file) > > After aborting "ctl_conversationsdb -b" "cyr_dbtool show" didn't show any > duplicates. But the string command showed more occurrences than cyr_dbtool > did (i used an hash value (d8087b76083ec5e7) I found in the output of > cyr_dbtool > > > # cyr_dbtool /path-to/userid.conversations skiplist show | grep > d8087b76083ec5e7 | wc -l > 3 > # cyr_dbtool /path-to/userid.conversations skiplist show | grep > Bd8087b76083ec5e7 | wc -l > 1 > # strings /path-to/userid.conversations | grep d8087b76083ec5e7 | wc -l > 183 > # strings /path-to/userid.conversations | grep Bd8087b76083ec5e7 | wc -l > 91 > > ctl_conversationsdb -d did a cleanup because I did abort the > "ctl_conversationsdb -b" process > > skiplist recovery /path-to/userid.conversations: found partial txn, > not replaying > skiplist: recovered /path-to/userid.conversations (0 records, 144 > bytes) in 0 seconds > skiplist: checkpointed /path-to/userid.conversations (0 records, 144 > bytes) in 0.010 sec > > > I did run ctl_conversationsd -b with strace and discovered that the inbox > (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache cyrus.O_RDWR ) of that user > was opened over and over again, no other folder for that user was accessed. > For other users the inbox was only opened once and then the other > folders followed. > So not stopping "ctl_conversationsd -b", it would have run till my > filesystem was > full. > I guess something about that inbox is confusing it, and making it redo it over and over? Very curious > > Bron, does this sound like anything you've seen before, that maybe > > got fixed on master but not backported to 3.0? > > > >> Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock > >> user.UserID for 1.5 seconds" > >> every few seconds. > > > > This message is logged when the lock is released, if that lock had > > been held for >1s. It's reporting that it held the lock for longer > > than it would like to have, but at least you know the lock has been > > released. I think this is telling you that user-visible performance > > on the mailbox would have been impacted while this was occurring > > (because imapd/lmtpd wouldn't be able to access their mailbox while > > ctl_conversationsdb was holding those locks), but also tells you > > that ctl_conversationsdb was doing the right thing, and not just > > holding a single lock for the entire job. > > > > I am not sure if any other processes had a chance to acquire a lock. If they'd tried, they would have succeeded (by blocking until the lock was theirs, during the gaps where ctl_conversationsdb dropped and reacquired the lock), but maybe nothing else wanted it. > > Though, that raises the question: was the user accessing the mailbox > > while the rebuild was in progress? I wonder if their client is > > doing something funny and tripping things up? > > We did this while the server was accessible (didn't want to have > several hour downtime for ~44000 users) Fair enough! > But as I did my recent testing on a test-server without user access, > only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during > the conversation rebuild mentioned in this mail. So I don't think that > user access is part of the problem. Thanks for testing that, good to rule it out as a culprit > So for me it looks like we have two problems: > > 1. multiple entries for the same key in skiplist db files. > As skiplist is a Key/Value store this should not be possible > to have duplicate keys? I don't know the internals of our database formats well, but I _think_ maybe it writes new key/value pairs by appending to the end of the database, and then updating the skiplist structure to point to the newer version instead of the older version. If this is correct, then if it's rewriting the keys over and over for some reason, I would expect the raw file to contain a lot of old, unlinked records for the same key, but which aren't visible over the API. I believe there's a process that runs occasionally that rewrites these databases with only the "live" keys and gets back the wasted space? > Is twoskip a better alternative? This is before my time, but twoskip was written to solve problems FastMail experienced with using skiplist with conversations ("post-crash recovery was too slow"). So if there's a pathological case in the more recent conversations code that trips a bug in skiplist, we probably wouldn't see it. So, I'd recommend using twoskip for conversationsdb just based on that :) > 2. Why didn't ctl_conversationsdb continue to process > the next mailbox but re-did the INBOX / the last mailbox It kinda sounds to me like it considered the operation to be unsuccessful, and tried again? Not sure what would cause this. Is there anything interesting in syslog? > I will try to run a debugger on ctl_conversationsdb and > will test with twoskip as conversations db format Thanks, it'd be very interesting to see if this issue reproduces with the twoskip format! > I did notice one other problem, recreating the conversations db > on the replica confused the sync protocol as the rebuild did increase > the modseq. Do you mean to say that this user's conversations db could be rebuilt successfully on the replica? Or was this with one of the "good" users? > Many logs like "record mismatch with replica: %s more recent on > replica" on the sync client > and "higher modseq on replica" on the sync server > > is this intended? If a conversation needs to be split into multiple smaller conversations (conversations_max_thread setting), then modseqs also need to be bumped, so this is not entirely unexpected Cheers, ellie From miguel at prodemge.gov.br Wed Jan 23 10:46:37 2019 From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira) Date: Wed, 23 Jan 2019 13:46:37 -0200 Subject: =?utf-8?q?Re=3A?= Mailboxes with slow access Message-ID: <12ac-5c488c80-b-4b99fb00@54616059> Hi Ellie, Thanks for answers, I'm gonna check how much memory the imap process is using like you suggest. Any other update about this issue I'll post here. Regards -- Miguel Moreira Gerente DPR/SRE/GSR - Ger?ncia de Servi?os de Rede +55(31)3339-1401 PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o. Em Domingo, Janeiro 20, 2019 23:33 -02, ellie timoney escreveu:?Hi Miguel,?On Sat, Jan 19, 2019, at 8:20 AM, Miguel Mucio Santos Moreira wrote:Hi folks!?I've had a problem with some huge mailboxes, with a lot of folders and subfolders. Problably thousand of folders.The access on these mailboxes is too slow, if you change from INBOX to another folder called CYRUS, even if CYRUS folder has just 4 messages for example the access is slow.?This sounds like you don't have enough RAM to hold the metadata for these mailboxes in memory, and so the system is swapping while accessing them.?You could find the imapd process that is servicing this connection and look at its virtual memory size, and compare that to the amount of physical RAM in your system, to see whether this is the case.?Or, if you post up the size of these mailboxes' cyrus.header, cyrus.index, cyrus.annotations, cyrus.cache (...) files and the size of their [user].conversations file (if you have conversations enabled), then maybe someone will be able to suggest a RAM size that would support this.?You mention being in a murder environment -- I'm not sure if you will need to increase the RAM on just the backend for these mailboxes, or the frontends as well. ?Maybe someone running a large murder can chime in with how they provision their servers??Sorry I don't have specific numbers for you, but hopefully this points you in the right direction!?I was trying to find out? anything on internet about this issue and I read that a parameter maxword on imapd.conf? could be the solution.Nowadays in our cyrus-imap murder all backend servers and frontend servers have the maxword parameter set to default value which is 128kb.Based on that information, I'd like to know if I increase this parameter to a high value like 1Mb for instance it could fix the issue about slow access and which considerations about memory, disk and cpu resources I must take care if I set up this value in my environment.Another question is, this value must be setting in all backends and frontends servers??imapd.conf(5) says:? ? ? ? ? maxword: 131072????????????? Maximum size of a single word for the parser.? Default 128k?This setting won't have any effect on performance at all. ?It's just an upper limit for the IMAP parser -- if some IMAP client tries to send a single word that is longer than this, the imapd process serving their connection will exit immediately with a "word too long" error rather than trying to keep parsing. ?Normal traffic won't hit this limit, it's just protecting itself from (possibly malicious) garbage. ?The default should be never need changing really.?Hope this helps,?ellie -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprice at gibb.co.za Thu Jan 24 04:51:45 2019 From: nprice at gibb.co.za (Neil Price) Date: Thu, 24 Jan 2019 11:51:45 +0200 Subject: What do these mean in cyrus.header? Message-ID: <602e8f9d-fde8-4ee2-9254-77d197c1a1f0@gibb.co.za> /nonjunk $has_cal $Forwarded $MDNSent junk $label3 $label2 icc-forward $label1 $label4 $label5/ // -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Thu Jan 24 05:09:42 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 24 Jan 2019 11:09:42 +0100 Subject: What do these mean in cyrus.header? In-Reply-To: <602e8f9d-fde8-4ee2-9254-77d197c1a1f0@gibb.co.za> References: <602e8f9d-fde8-4ee2-9254-77d197c1a1f0@gibb.co.za> Message-ID: They are IMAP flags that have been used for messages in that mailbox. --On 24. Januar 2019 um 11:51:45 +0200 Neil Price wrote: > /nonjunk $has_cal $Forwarded $MDNSent junk $label3 $label2 icc-forward > $label1 $label4 $label5/ > // -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From michael.menge at zdv.uni-tuebingen.de Thu Jan 24 05:12:37 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Thu, 24 Jan 2019 11:12:37 +0100 Subject: prblems rebuilding conversations db In-Reply-To: <1548108964.1676215.1640323480.73A4EE8F@webmail.messagingengine.com> References: <20190116121247.Horde.-zO1HRoBQLXFDoDXDRuG40r@webmail.uni-tuebingen.de> <1548037219.3832722.1639587944.75697E74@webmail.messagingengine.com> <20190121143207.Horde.oR1nN76B2p9NRilOsUDDwg4@webmail.uni-tuebingen.de> <1548108964.1676215.1640323480.73A4EE8F@webmail.messagingengine.com> Message-ID: <20190124111237.Horde.uNWL834sydcRBNO8a1NaDpR@webmail.uni-tuebingen.de> Hi Quoting ellie timoney : ... > On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote: ... >> Quoting ellie timoney : ... >> > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote: >> >> Hi, >> >> >> >> because conversations db seems to be required for search indexes, I >> >> enabled this option >> >> on our production servers today and tried to rebuild the conversations >> >> db for all users with >> >> >> >> ctl_conversationsdb -v -b UID >> >> >> >> For most users this did take less than a second. But for some users >> >> this process would >> >> not finish. I did kill one process after about 30 Minutes (most others >> >> after 3 Minutes). >> >> The UserID.conversations has grown over 2 GB (the mailbox itself has >> >> only ~700 MB of mails, >> >> and the conversations files from finnished rebuilds are less then 20 MB) ... >> I did run ctl_conversationsd -b with strace and discovered that the inbox >> (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache cyrus.O_RDWR ) >> of that user >> was opened over and over again, no other folder for that user was accessed. >> For other users the inbox was only opened once and then the other >> folders followed. >> So not stopping "ctl_conversationsd -b", it would have run till my >> filesystem was >> full. >> > > I guess something about that inbox is confusing it, and making it > redo it over and over? Very curious > ... >> > Though, that raises the question: was the user accessing the mailbox >> > while the rebuild was in progress? I wonder if their client is >> > doing something funny and tripping things up? >> >> We did this while the server was accessible (didn't want to have >> several hour downtime for ~44000 users) > > Fair enough! > >> But as I did my recent testing on a test-server without user access, >> only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during >> the conversation rebuild mentioned in this mail. So I don't think that >> user access is part of the problem. > > Thanks for testing that, good to rule it out as a culprit > >> So for me it looks like we have two problems: >> >> 1. multiple entries for the same key in skiplist db files. >> As skiplist is a Key/Value store this should not be possible >> to have duplicate keys? > > I don't know the internals of our database formats well, but I > _think_ maybe it writes new key/value pairs by appending to the end > of the database, and then updating the skiplist structure to point > to the newer version instead of the older version. If this is > correct, then if it's rewriting the keys over and over for some > reason, I would expect the raw file to contain a lot of old, > unlinked records for the same key, but which aren't visible over the > API. I believe there's a process that runs occasionally that > rewrites these databases with only the "live" keys and gets back the > wasted space? > I did discover, because of the loop (see below) the point of database commit was not reached, so I guess that could explain that there was no duplicated key cleanup done. >> Is twoskip a better alternative? > > This is before my time, but twoskip was written to solve problems > FastMail experienced with using skiplist with conversations > ("post-crash recovery was too slow"). So if there's a pathological > case in the more recent conversations code that trips a bug in > skiplist, we probably wouldn't see it. So, I'd recommend using > twoskip for conversationsdb just based on that :) Same problem with twoskip, but as I said above, the commit point was not reached. > >> 2. Why didn't ctl_conversationsdb continue to process >> the next mailbox but re-did the INBOX / the last mailbox > > It kinda sounds to me like it considered the operation to be > unsuccessful, and tried again? Not sure what would cause this. Is > there anything interesting in syslog? > >> I will try to run a debugger on ctl_conversationsdb and >> will test with twoskip as conversations db format > The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189) For the problematic mailbox that I tested, for every message record->cid was NULLCONVERSATION, so mailbox_cacherecord, message_update_conversations and mailbox_rewrite_index_record where called, each returned 0. After iterating trough all messages in the mailbox count was > 0, and r==0, so the while condition (!r && count) was true for the next run. At this point record->cid was still NULLCONVERSATION for every message, which I guess should not be the case. > Thanks, it'd be very interesting to see if this issue reproduces > with the twoskip format! > Yes, it did. >> I did notice one other problem, recreating the conversations db >> on the replica confused the sync protocol as the rebuild did increase >> the modseq. > > Do you mean to say that this user's conversations db could be > rebuilt successfully on the replica? Or was this with one of the > "good" users? > I did activated the conversation db first on the replica (to watch the performance impact and time needed) and did the rebuild with the -r flag. I didn't notice an endless loop on the replic, but remember the creation of the conversation db failed for some users and I had no good way to know for which user the process was successful and for which it failed I decided to run a single process for each user and check the return value. I will test if the -r flag changes the rebuild on my test-server. ... -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From michael.menge at zdv.uni-tuebingen.de Thu Jan 24 12:02:32 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Thu, 24 Jan 2019 18:02:32 +0100 Subject: another problem with conversations db Message-ID: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> Hi, I have discovered an other problem with the conversations db: Thousends of lines with "IOERROR: conversations_audit on load:" and "IOERROR: conversations_audit on store:" A look at the source code shows that these errors are logged after "_sanity_check_counts" is called. The log level LOG_ERR and the prefix IOERROR indicate that I have a serious problem. Do I? This problem occurred for accounts where the rebuild of the conversations db was successful. I don't want to rebuild the conversations db every few days. Any help is appreciated. Kind regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From brong at fastmail.fm Thu Jan 24 15:45:11 2019 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 24 Jan 2019 15:45:11 -0500 Subject: another problem with conversations db In-Reply-To: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> Message-ID: <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> Sorry I haven't been following along with this earlier. Can you post your imapd.conf and cyrus.conf as well as let her know if you run anything which directly messes with files on disk. Also, what operating system is this on and what Cyrus version? Bron. On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > Hi, > > I have discovered an other problem with the conversations db: > > Thousends of lines with "IOERROR: conversations_audit on load:" and > "IOERROR: conversations_audit on store:" > A look at the source code shows that these errors are logged after > "_sanity_check_counts" is called. > The log level LOG_ERR and the prefix IOERROR indicate that I have a > serious problem. Do I? > > This problem occurred for accounts where the rebuild of the > conversations db was successful. > > I don't want to rebuild the conversations db every few days. > > Any help is appreciated. > > Kind regards > > > Michael Menge > > > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.menge at zdv.uni-tuebingen.de Fri Jan 25 03:55:38 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Fri, 25 Jan 2019 09:55:38 +0100 Subject: another problem with conversations db In-Reply-To: <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> Message-ID: <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> Hi Bron Quoting Bron Gondwana : > Sorry I haven't been following along with this earlier. Can you post > your imapd.conf and cyrus.conf as well as let her know if you run > anything which directly messes with files on disk. Thanks for looking into it. I have attached the backend and replica configs. There should be nothing messing with the files on disk directly. Some times we need to restore mails from file based backup (bacula) followed by a reconstruct but this was not the case this time. > > Also, what operating system is this on and what Cyrus version? > We are running a cyrus 3.0.8 compiled with the following Options (./configure --enable-murder --enable-http --enable-calalarmd --enable-replication --enable-backup --enable-idled --enable-autocreate CFLAGS="-fPIC -g") in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > Bron. > > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: >> Hi, >> >> I have discovered an other problem with the conversations db: >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and >> "IOERROR: conversations_audit on store:" >> A look at the source code shows that these errors are logged after >> "_sanity_check_counts" is called. >> The log level LOG_ERR and the prefix IOERROR indicate that I have a >> serious problem. Do I? >> >> This problem occurred for accounts where the rebuild of the >> conversations db was successful. >> >> I don't want to rebuild the conversations db every few days. >> >> Any help is appreciated. >> >> Kind regards >> >> >> Michael Menge >> >> >> >> >> >> -------------------------------------------------------------------------------- >> M.Menge Tel.: (49) 7071/29-70316 >> Universit?t T?bingen Fax.: (49) 7071/29-5912 >> Zentrum f?r Datenverarbeitung mail: >> michael.menge at zdv.uni-tuebingen.de >> W?chterstra?e 76 >> 72074 T?bingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > -- > Bron Gondwana > brong at fastmail.fm -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- # Configuration for Backend/Failover Instance # Template MD5SUM: b5b04095d552bb1cbc2b178f7edfd1e2 conf/imapd_master.template servername: ma01.mail.localhost configdirectory: /srv/cyrus-be partition-default: /srv/cyrus-be partition-ssd: /srv/cyrus-be/ssd-part metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part metapartition_files: header index cache expunge squat annotations lock dav archivecache archivepartition-ssd: /srv/cyrus-hdd-be/archive/ssd-part archive_enabled: 1 archive_days: 31 archive_maxsize: 10240 proc_path: /srv/tmpfs/proc-be mboxname_lockpath: /srv/tmpfs/lock-be defaultpartition: ssd admins: YYYY YYYY mupdate_server: mupdate.mail.localhost mupdate_port: 3905 # mupdate_authname verwenden nicht mupdate_username # sonst wird login als root/cyrus versucht #mupdate_username: YYYY mupdate_authname: YYYY mupdate_password: XXXX proxy_authname: YYYY proxy_password: XXXX proxyservers: YYYY allowusermoves: 1 allowallsubscribe: 1 sync_host: sl01.mail.localhost sync_authname: YYYY sync_password: XXXX sync_port: 2005 guid_mode: sha1 sync_log: 1 sync_shutdown_file: /srv/cyrus-be/sync/shutdown sievedir: /srv/cyrus-be/sieve sieve_extensions: fileinto reject vacation imapflags notify include envelope body relational regex subaddress copy sieve_maxscriptsize: 150 sasl_pwcheck_method: saslauthd sasl_mech_list: plain login allowanonymouslogin: no reject8bit: no quotawarn: 90 timeout: 45 poptimeout: 10 dracinterval: 0 drachost: localhost lmtp_over_quota_perm_failure: 1 altnamespace: 1 #flushseenstate: 1 unixhierarchysep: 1 hashimapspool: 1 fulldirhash: 1 duplicatesuppression: 0 expunge_mode: delayed delete_mode: delayed deletedprefix: DELETED anysievefolder: 1 #annotation_db: skiplist #duplicate_db: skiplist #mboxlist_db: skiplist #ptscache_db: skiplist quota_db: quotalegacy #seenstate_db: skiplist subscriptions_db: flat xlist-sent: Mail.sent xlist-trash: Mail.trash xlist-drafts: Mail.drafts xlist-junk: Mail.v-spam xlist-spam: Mail.v-spam specialusealways: 1 syslog_prefix: be # https://bugzilla.cyrusimap.org/show_bug.cgi?id=3850 # Vermutlich gefixed #suppress_capabilities: LIST-EXTENDED tls_server_cert: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_server_key: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_client_ca_file: /etc/pki/tls/certs/ca-bundle.crt tls_ciphers: EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!3DES:!IDEA tls_prefer_server_ciphers: 1 # ?nderungen 27.06.2018 reverseacls: 1 search_engine: squat delete_unsubscribe: 1 statuscache: 1 #tlscache_db deprecated #tls_sessions_db: skiplist # ?nderung 11.01.2019 # conversations: 1 conversations_expire_days: 90 beimap_auditlog: 1 beimaps_auditlog: 1 bepop3_auditlog: 1 bepop3s_auditlog: 1 belmtp_auditlog: 1 bedelprune_auditlog: 1 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cyrus_be.conf URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cyrus_re.conf URL: -------------- next part -------------- # Configuration for Slave (Replica) Instance # Template MD5SUM: fa40ce3a963f578f143005302c9edea4 conf/imapd_slave.template servername: sl08.mail.localhost configdirectory: /srv/cyrus-re partition-default: /srv/cyrus-re partition-ssd: /srv/cyrus-re/ssd-part metapartition-ssd: /srv/cyrus-ssd-re/meta/ssd-part metapartition_files: header index cache expunge squat annotations lock dav archivecache archivepartition-ssd: /srv/cyrus-hdd-re/archive/ssd-part archive_enabled: 1 archive_days: 31 archive_maxsize: 10240 proc_path: /srv/tmpfs/proc-re mboxname_lockpath: /srv/tmpfs/lock-re defaultpartition: ssd admins: YYYY YYYY allowusermoves: 1 allowallsubscribe: 1 proxy_authname: YYYY proxy_password: XXXX proxyservers: YYYY sievedir: /srv/cyrus-re/sieve sieve_extensions: fileinto reject vacation imapflags notify include envelope body relational regex subaddress copy sieve_maxscriptsize: 150 sasl_pwcheck_method: saslauthd sasl_mech_list: plain login allowanonymouslogin: no reject8bit: no quotawarn: 90 timeout: 45 poptimeout: 10 dracinterval: 0 drachost: localhost lmtp_over_quota_perm_failure: 1 altnamespace: 1 #flushseenstate: 1 unixhierarchysep: 1 hashimapspool: 1 fulldirhash: 1 duplicatesuppression: 0 expunge_mode: delayed delete_mode: delayed deletedprefix: DELETED anysievefolder: 1 #annotation_db: skiplist #duplicate_db: skiplist #mboxlist_db: skiplist #ptscache_db: skiplist quota_db: quotalegacy #seenstate_db: skiplist subscriptions_db: flat syslog_prefix: re # https://bugzilla.cyrusimap.org/show_bug.cgi?id=3850 # Vermutlich gefixed #suppress_capabilities: LIST-EXTENDED tls_server_cert: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_server_key: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_client_ca_file: /etc/pki/tls/certs/ca-bundle.crt tls_ciphers: EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!3DES:!IDEA tls_prefer_server_ciphers: 1 #tlscache_db deprecated #tls_sessions_db: skiplist # ?nderungen 27.06.2018 reverseacls: 1 statuscache: 1 # ?nderung 10.01.2019 search_engine: squat conversations: 1 conversations_expire_days: 90 reimap_auditlog: 1 reimaps_auditlog: 1 repop3_auditlog: 1 repop3s_auditlog: 1 relmtp_auditlog: 1 redelprune_auditlog: 1 From brong at fastmail.fm Fri Jan 25 05:55:14 2019 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 25 Jan 2019 05:55:14 -0500 Subject: another problem with conversations db In-Reply-To: <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> Message-ID: <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > Hi Bron > > > Quoting Bron Gondwana : > > > Sorry I haven't been following along with this earlier. Can you post > > your imapd.conf and cyrus.conf as well as let her know if you run > > anything which directly messes with files on disk. > > Thanks for looking into it. I have attached the backend and replica > configs. There should be nothing messing with the files on disk > directly. Some times we need to restore mails from file based > backup (bacula) followed by a reconstruct but this was not the > case this time. Do you always rebuild the conversations db after doing the reconstruct? That will be necessary now. We switched to doing all our restores using IMAP append a while back so we're never fiddling the file system under Cyrus. > > > > Also, what operating system is this on and what Cyrus version? > > > > We are running a cyrus 3.0.8 compiled with the following Options > (./configure --enable-murder --enable-http --enable-calalarmd > --enable-replication --enable-backup --enable-idled > --enable-autocreate CFLAGS="-fPIC -g") > in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. They should be fine. I'll have a read of the config when I'm at a real computer. > > > Bron. > > > > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> Hi, > >> > >> I have discovered an other problem with the conversations db: > >> > >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> "IOERROR: conversations_audit on store:" > >> A look at the source code shows that these errors are logged after > >> "_sanity_check_counts" is called. > >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> serious problem. Do I? > >> > >> This problem occurred for accounts where the rebuild of the > >> conversations db was successful. > >> > >> I don't want to rebuild the conversations db every few days. > >> > >> Any help is appreciated. > >> > >> Kind regards > >> > >> > >> Michael Menge > >> > >> > >> > >> > >> > >> -------------------------------------------------------------------------------- > >> M.Menge Tel.: (49) 7071/29-70316 > >> Universit?t T?bingen Fax.: (49) 7071/29-5912 > >> Zentrum f?r Datenverarbeitung mail: > >> michael.menge at zdv.uni-tuebingen.de > >> W?chterstra?e 76 > >> 72074 T?bingen > >> > >> ---- > >> Cyrus Home Page: http://www.cyrusimap.org/ > >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > >> To Unsubscribe: > >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > -- > > Bron Gondwana > > brong at fastmail.fm > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * imapd_be.conf > * cyrus_be.conf > * cyrus_re.conf > * imapd_re.conf -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.menge at zdv.uni-tuebingen.de Fri Jan 25 06:25:53 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Fri, 25 Jan 2019 12:25:53 +0100 Subject: another problem with conversations db In-Reply-To: <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> Message-ID: <20190125122553.Horde.tHssU9Ti123qWJ0cZj5sMpV@webmail.uni-tuebingen.de> Hi, Quoting Bron Gondwana : > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: >> Hi Bron >> >> >> Quoting Bron Gondwana : >> >> > Sorry I haven't been following along with this earlier. Can you post >> > your imapd.conf and cyrus.conf as well as let her know if you run >> > anything which directly messes with files on disk. >> >> Thanks for looking into it. I have attached the backend and replica >> configs. There should be nothing messing with the files on disk >> directly. Some times we need to restore mails from file based >> backup (bacula) followed by a reconstruct but this was not the >> case this time. > > Do you always rebuild the conversations db after doing the > reconstruct? That will be necessary now. We switched to doing all > our restores using IMAP append a while back so we're never fiddling > the file system under Cyrus. > We didn't have to restore any mails from backup since we enabled conversation db a few weeks ago. It is good to know that the rebuild is necessary, but shouldn't reconstruct also update the conversation db if it re-appends the message? At least a hint should be put in the reconstruct man page, like the one for "quota -f" >> > >> > Also, what operating system is this on and what Cyrus version? >> > >> >> We are running a cyrus 3.0.8 compiled with the following Options >> (./configure --enable-murder --enable-http --enable-calalarmd >> --enable-replication --enable-backup --enable-idled >> --enable-autocreate CFLAGS="-fPIC -g") >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > They should be fine. I'll have a read of the config when I'm at a > real computer. > >> >> > Bron. >> > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: >> >> Hi, >> >> >> >> I have discovered an other problem with the conversations db: >> >> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and >> >> "IOERROR: conversations_audit on store:" >> >> A look at the source code shows that these errors are logged after >> >> "_sanity_check_counts" is called. >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a >> >> serious problem. Do I? >> >> >> >> This problem occurred for accounts where the rebuild of the >> >> conversations db was successful. >> >> >> >> I don't want to rebuild the conversations db every few days. >> >> >> >> Any help is appreciated. >> >> >> >> Kind regards >> >> >> >> >> >> Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From brong at fastmail.fm Fri Jan 25 20:27:17 2019 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 25 Jan 2019 20:27:17 -0500 Subject: another problem with conversations db In-Reply-To: <20190125122553.Horde.tHssU9Ti123qWJ0cZj5sMpV@webmail.uni-tuebingen.de> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> <20190125122553.Horde.tHssU9Ti123qWJ0cZj5sMpV@webmail.uni-tuebingen.de> Message-ID: <782f66eb-c04c-4cc0-9b2c-638e57354245@beta.fastmail.com> Yes, updating reconstruct is on our roadmap. The biggest problem is that there are multiple file types across mailboxes and conversations and the changes can't be atomic, so if reconstruct discovers something partially done, it can't tell if the conversations were correct. The longer term grand plan is to reduce the number of db types and make conversations updates able to be atomic. Cheers, Bron. On Fri, Jan 25, 2019, at 22:30, Michael Menge wrote: > Hi, > > > Quoting Bron Gondwana : > > > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > >> Hi Bron > >> > >> > >> Quoting Bron Gondwana : > >> > >> > Sorry I haven't been following along with this earlier. Can you post > >> > your imapd.conf and cyrus.conf as well as let her know if you run > >> > anything which directly messes with files on disk. > >> > >> Thanks for looking into it. I have attached the backend and replica > >> configs. There should be nothing messing with the files on disk > >> directly. Some times we need to restore mails from file based > >> backup (bacula) followed by a reconstruct but this was not the > >> case this time. > > > > Do you always rebuild the conversations db after doing the > > reconstruct? That will be necessary now. We switched to doing all > > our restores using IMAP append a while back so we're never fiddling > > the file system under Cyrus. > > > > We didn't have to restore any mails from backup since we enabled > conversation db > a few weeks ago. It is good to know that the rebuild is necessary, but > shouldn't > reconstruct also update the conversation db if it re-appends the message? > At least a hint should be put in the reconstruct man page, like the > one for "quota -f" > > > >> > > >> > Also, what operating system is this on and what Cyrus version? > >> > > >> > >> We are running a cyrus 3.0.8 compiled with the following Options > >> (./configure --enable-murder --enable-http --enable-calalarmd > >> --enable-replication --enable-backup --enable-idled > >> --enable-autocreate CFLAGS="-fPIC -g") > >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > > > They should be fine. I'll have a read of the config when I'm at a > > real computer. > > > >> > >> > Bron. > >> > > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> >> Hi, > >> >> > >> >> I have discovered an other problem with the conversations db: > >> >> > >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> >> "IOERROR: conversations_audit on store:" > >> >> A look at the source code shows that these errors are logged after > >> >> "_sanity_check_counts" is called. > >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> >> serious problem. Do I? > >> >> > >> >> This problem occurred for accounts where the rebuild of the > >> >> conversations db was successful. > >> >> > >> >> I don't want to rebuild the conversations db every few days. > >> >> > >> >> Any help is appreciated. > >> >> > >> >> Kind regards > >> >> > >> >> > >> >> Michael Menge > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.menge at zdv.uni-tuebingen.de Thu Jan 31 03:35:59 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Thu, 31 Jan 2019 09:35:59 +0100 Subject: another problem with conversations db In-Reply-To: <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de> <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com> <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de> <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com> Message-ID: <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de> Quoting Bron Gondwana : > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: >> Hi Bron >> >> >> Quoting Bron Gondwana : >> >> > Sorry I haven't been following along with this earlier. Can you post >> > your imapd.conf and cyrus.conf as well as let her know if you run >> > anything which directly messes with files on disk. >> >> Thanks for looking into it. I have attached the backend and replica >> configs. There should be nothing messing with the files on disk >> directly. Some times we need to restore mails from file based >> backup (bacula) followed by a reconstruct but this was not the >> case this time. > > Do you always rebuild the conversations db after doing the > reconstruct? That will be necessary now. We switched to doing all > our restores using IMAP append a while back so we're never fiddling > the file system under Cyrus. > >> > >> > Also, what operating system is this on and what Cyrus version? >> > >> >> We are running a cyrus 3.0.8 compiled with the following Options >> (./configure --enable-murder --enable-http --enable-calalarmd >> --enable-replication --enable-backup --enable-idled >> --enable-autocreate CFLAGS="-fPIC -g") >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > They should be fine. I'll have a read of the config when I'm at a > real computer. > did you find anything in the config that would explain the problems? >> >> > Bron. >> > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: >> >> Hi, >> >> >> >> I have discovered an other problem with the conversations db: >> >> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and >> >> "IOERROR: conversations_audit on store:" >> >> A look at the source code shows that these errors are logged after >> >> "_sanity_check_counts" is called. >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a >> >> serious problem. Do I? >> >> >> >> This problem occurred for accounts where the rebuild of the >> >> conversations db was successful. >> >> >> >> I don't want to rebuild the conversations db every few days. >> >> >> >> Any help is appreciated. >> >> >> >> Kind regards >> >> >> >> >> >> Michael Menge >> >> >> >> >> >> >> >> >> >> >> >> >> -------------------------------------------------------------------------------- >> >> M.Menge Tel.: (49) 7071/29-70316 >> >> Universit?t T?bingen Fax.: (49) 7071/29-5912 >> >> Zentrum f?r Datenverarbeitung mail: >> >> michael.menge at zdv.uni-tuebingen.de >> >> W?chterstra?e 76 >> >> 72074 T?bingen >> >> >> >> ---- >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> >> To Unsubscribe: >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> > >> > -- >> > Bron Gondwana >> > brong at fastmail.fm >> >> >> >> -------------------------------------------------------------------------------- >> M.Menge Tel.: (49) 7071/29-70316 >> Universit?t T?bingen Fax.: (49) 7071/29-5912 >> Zentrum f?r Datenverarbeitung mail: >> michael.menge at zdv.uni-tuebingen.de >> W?chterstra?e 76 >> 72074 T?bingen >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> *Attachments:* >> * imapd_be.conf >> * cyrus_be.conf >> * cyrus_re.conf >> * imapd_re.conf > > -- > Bron Gondwana > brong at fastmail.fm -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen