From egoitz at sarenet.es Thu Jan 3 07:27:26 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 13:27:26 +0100 Subject: www.cyrusimap.org down? Message-ID: Good afternoon, I'm suffering connectivity issues to Cyrus imap web site. Perhaps some kind of problem at hosting side?. It takes from yesterday... do we know something about when will it be up again?. Cheers! -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 08:53:30 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 14:53:30 +0100 Subject: Question for upgrading In-Reply-To: <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> Message-ID: <4d260681b7e89b934237a64ade5d4502@sarenet.es> Hi Nic, Sorry for the delay answering... I have been ill and later on holiday, with very very very few time to answer to nothing when coming from illness. Sorry then... My idea really, was to upgrade to 3.0... So I'm planning: - Upgrade in place from 2.3 to 2.4. - Continue giving the service... - Later setup a 3.0 slave.... all of this WITHOUT doing a reconstruct -V max..... as it seems in 2.4 all works (unexpunge command and so) without it... should say too, it seems the own sync client performs database upgrade.... Jan 3 14:46:02 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Alberto (10 -> 12) Jan 3 14:46:06 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Comerciales (10 -> 12) Jan 3 14:46:15 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.Instalaciones-staff (10 -> 12) Jan 3 14:46:25 mx8c sync_client[17186]: Index upgrade: mydomain.com!user.mytestuser.Sarenet-staff.RRHH (10 -> 12) It seems to anyway be happening.... - When the 3.0 slave is totally replicated from the 2.4 mater... turn the 3.0 as mater, and take a copy of it becoming it slave..... This way I should have all the platform upgraded without downtimes.... How do you see Nic?. I think it should work... I'm still doing checks anyway..... Cheers!! Thanks a lot for your time!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [2] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 15-12-2018 17:05, Nic Bernstein escribi?: > On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: > >> I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem is that this reconstruct command, takes ages and I'm not able to keep the service offline so many time. So I have been thinking in the following scenario : > > Egoitz, > As long as you've followed all of the various steps needed to account for version changes, as outlined in the Release Notes for _all_ intermediary releases, then the last step should be the updating of the indexes. Rather than running "reconstruct -V max" on all mailboxes at once, simply run it on subsets. We've done this on large installations without ill effect. You can have the service on line during the reconstruct, and, as you note, have most all functionality present. > > We build a list of users (mboxlist.txt), and then run a cron job, during off-peak hours, using the 'parallel' command like so (from 'crontab -e -u cyrus'): > >> ### >> ## Start a batch of recursive reconstruct jobs at 7PM and discontinue >> ## at 6:53AM. >> 0 19 * * * parallel -j 5 --resume --joblog /var/lib/imap/reconstruct.log cyrus reconstruct -G -V max -r -u {} < /var/lib/imap/mboxlist.txt >> 53 06 * * * kill -TERM `ps ax | grep [p]arallel | awk '{print $1}'` > Cheers, > -nic > > -- > Nic Bernstein nic at onlight.com > Onlight Inc. www.onlight.com [1] > 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 > Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 Links: ------ [1] http://www.onlight.com [2] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 09:55:28 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 15:55:28 +0100 Subject: Xapian searches in Cyrus 3.0 Message-ID: Good afternoon, I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Jan 3 10:32:16 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 16:32:16 +0100 Subject: Question about manual replication (-u ) Message-ID: <9797216371bb00cdb59351370d5607d2@sarenet.es> Good afternoon, Is it possible to launch several instances of "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. Doing it just one mailbox at a time takes ages.... It would help me a lot, the fact of parallelizing and have no disk bottleneck issues.... I think it should be possible... isn't it?. Perhaps it just allowed between same version in source and dest?. Or can be done for instance too, with a 2.4 as master to 3.0 slave?. Cheers!! -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Thu Jan 3 11:00:30 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Thu, 03 Jan 2019 16:00:30 +0000 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: References: Message-ID: <820622b2dae3124f681406aee6b0c4e065b254ef.camel@aegee.org> Hello Egoitz, concerning Xapian see discussion at https://github.com/cyrusimap/cyrus-imapd/issues/2234 . I am still struggling how to verify that for new mails the search information is append, and that for searches this information is utilized. Regards ????? On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote: > Good afternoon, > > I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. > > Best regards, > -- > > Egoitz Aurrekoetxea > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. From egoitz at sarenet.es Thu Jan 3 11:09:37 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 17:09:37 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <820622b2dae3124f681406aee6b0c4e065b254ef.camel@aegee.org> References: <820622b2dae3124f681406aee6b0c4e065b254ef.camel@aegee.org> Message-ID: Hi Dilyan, I suppose that should be easy. Run Squatter in rolling mode and send you a message in a testing mailbox where no other emails exist. Then, try to find that word :).. shouldn't it?. Or else, send a mailbox with a big word to search and see if it appears in a search... by the way... when you receive that mail, Squatter should be triggered (when in rolling mode) to index that last email... Hope it helps! Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 03-01-2019 17:00, ????? ???????? escribi?: > Hello Egoitz, > > concerning Xapian see discussion at https://github.com/cyrusimap/cyrus-imapd/issues/2234 . > > I am still struggling how to verify that for new mails the search information is append, and that for searches this > information is utilized. > > Regards > ????? > > On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote: > >> Good afternoon, >> >> I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. >> >> Best regards, >> -- >> >> Egoitz Aurrekoetxea >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol?gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es >> www.sarenet.es [1] >> >> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Thu Jan 3 11:53:55 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Thu, 03 Jan 2019 16:53:55 +0000 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: References: <820622b2dae3124f681406aee6b0c4e065b254ef.camel@aegee.org> Message-ID: <18abeb38a4c4b2df838562e176a0528d84961e67.camel@aegee.org> Hello Egoitz, my understanding is, that search always succeeds; indexed search is just faster. Cf. docsrc/imap/reference/admin/sop/sqatter.rst . Triggered squatter does not yet mean, that it stores some results on the hard disk. Regards ????? On Thu, 2019-01-03 at 17:09 +0100, Egoitz Aurrekoetxea wrote: > Hi Dilyan, > > I suppose that should be easy. Run Squatter in rolling mode and send you a message in a testing mailbox where no other emails exist. Then, try to find that word :).. shouldn't it?. Or else, send a mailbox with a big word to search and see if it appears in a search... by the way... when you receive that mail, Squatter should be triggered (when in rolling mode) to index that last email... > > Hope it helps! > Cheers! > --- > > Egoitz Aurrekoetxea > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 03-01-2019 17:00, ????? ???????? escribi?: > > Hello Egoitz, > > > > concerning Xapian see discussion at https://github.com/cyrusimap/cyrus-imapd/issues/2234 . > > > > I am still struggling how to verify that for new mails the search information is append, and that for searches this > > information is utilized. > > > > Regards > > ????? > > > > On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote: > > > Good afternoon, > > > > > > I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. > > > > > > Best regards, > > > -- > > > > > > Egoitz Aurrekoetxea > > > Departamento de sistemas > > > 944 209 470 > > > Parque Tecnol?gico. Edificio 103 > > > 48170 Zamudio (Bizkaia) > > > egoitz at sarenet.es > > > www.sarenet.es > > > > > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. From egoitz at sarenet.es Thu Jan 3 12:16:03 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 03 Jan 2019 18:16:03 +0100 Subject: Xapian searches in Cyrus 3.0 In-Reply-To: <18abeb38a4c4b2df838562e176a0528d84961e67.camel@aegee.org> References: <820622b2dae3124f681406aee6b0c4e065b254ef.camel@aegee.org> <18abeb38a4c4b2df838562e176a0528d84961e67.camel@aegee.org> Message-ID: Hi Dilyan, Yes, but I don't really know how Cyrus would behave if you have non indexed content on your mailbox and it's doing a indexed search. Perhaps it has a list of indexed emails and asummes (as seems obvious, they won't change?), and for the rest performs a non indexed search?. By the way, I assume that when you have a new mail on your mailbox and you see the following message due to the received new email : squatter[14957]: Xapian committed 1 updates in 0.004899 sec It should... have done something due to the mail delivered.... This is what I think... Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 03-01-2019 17:53, ????? ???????? escribi?: > Hello Egoitz, > > my understanding is, that search always succeeds; indexed search is just faster. Cf. > docsrc/imap/reference/admin/sop/sqatter.rst . > > Triggered squatter does not yet mean, that it stores some results on the hard disk. > > Regards > ????? > > On Thu, 2019-01-03 at 17:09 +0100, Egoitz Aurrekoetxea wrote: Hi Dilyan, > > I suppose that should be easy. Run Squatter in rolling mode and send you a message in a testing mailbox where no other emails exist. Then, try to find that word :).. shouldn't it?. Or else, send a mailbox with a big word to search and see if it appears in a search... by the way... when you receive that mail, Squatter should be triggered (when in rolling mode) to index that last email... > > Hope it helps! > Cheers! > --- > > Egoitz Aurrekoetxea > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 03-01-2019 17:00, ????? ???????? escribi?: Hello Egoitz, > > concerning Xapian see discussion at https://github.com/cyrusimap/cyrus-imapd/issues/2234 . > > I am still struggling how to verify that for new mails the search information is append, and that for searches this > information is utilized. > > Regards > ????? > > On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote: Good afternoon, > > I was planning to perform mailboxes squattering in rolling mode. Have you had some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so advantageous?. How your experiences had been in this sense, with this technology?. Is better to use a non Xapian system?. Just, let's say the old way of Squatter without Xapian?. > > Best regards, > -- > > Egoitz Aurrekoetxea > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 03:59:35 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 09:59:35 +0100 Subject: Xapian searches of the body of an email Message-ID: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Good morning, I have been testing Xapian searches. Have seen, it's not able to find strings inside the body of the email. If I set in imap.conf "search_fuzzy_always: 1" no content is displayed in the searches of a Roundcube stock webmail. If I remove that config value from imap.conf and restart services, then search results appear. Does Xapian not index the body of emails?. Does Xapian, just index the headers?. But this affirmation does not seem to be possible in my case too... as I have in the config "search_index_headers: no". I'm using the following config : conversations: 1 search_engine: xapian search_index_headers: no search_batchsize: 8192 defaultsearchtier: t1 t1searchpartition-default: /expert/search t1searchpartition-expert2: /expert2/search t1searchpartition-expert3: /expert3/search Could anyone help me mates?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Mon Jan 7 04:19:08 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 10:19:08 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: That sounds like the conversationsdb issue I was talking about. Have you tried these steps? ctl_conversationsdb -z USER ctl_conversationsdb -b USER > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 262 bytes Desc: not available URL: From egoitz at sarenet.es Mon Jan 7 08:51:02 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 14:51:02 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Hi mate! This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... Thanks a lot!! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > >> I have been testing Xapian searches. Have seen, it's not able to find >> strings inside the body of the email. If I set in imap.conf >> "search_fuzzy_always: 1" no content is displayed in the searches of a >> Roundcube stock webmail. If I remove that config value from imap.conf >> and restart services, then search results appear. Does Xapian not index >> the body of emails?. Does Xapian, just index the headers?. But this >> affirmation does not seem to be possible in my case too... as I have in >> the config "search_index_headers: no". >> >> I'm using the following config : >> >> conversations: 1 >> search_engine: xapian >> search_index_headers: no >> search_batchsize: 8192 >> defaultsearchtier: t1 >> t1searchpartition-default: /expert/search >> t1searchpartition-expert2: /expert2/search >> t1searchpartition-expert3: /expert3/search >> >> Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 09:05:52 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 15:05:52 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> And, by the way.... when using Squatter instead of Xapian as a search engine.... what do we really lost?. Just the fact of having a statistical worse results?. Is it Xapian faster than squat engine?. Sorry for having so many questions but... I suppose I don't have the implications of each one totally clear :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 14:51, Egoitz Aurrekoetxea escribi?: > Hi mate! > > This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... > > Thanks a lot!! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 09:41:59 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 15:41:59 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> Message-ID: <776585952e420b671ed40ed7495e88db@sarenet.es> Sorry for asking again.. ctl_conversationsdb -z and -b (with -r I assume for all users) should be run only on new user accounts or... periodically for any user account?. Or does squatter maintain too the conversations database?. Best regards, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 15:05, Egoitz Aurrekoetxea escribi?: > And, by the way.... > > when using Squatter instead of Xapian as a search engine.... what do we really lost?. Just the fact of having a statistical worse results?. Is it Xapian faster than squat engine?. > > Sorry for having so many questions but... I suppose I don't have the implications of each one totally clear :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 14:51, Egoitz Aurrekoetxea escribi?: > > Hi mate! > > This seems to take ages... I'm trying to figure the best way of implementing this and of clarifying concepts.... I'm running Squatter in rolling replication mode and exist the concept of conversations then. What is the exact role of each of them?. Squatter seems to index the mailbox but when something is not properly indexed instead of running Squatter you use ctl_conversations for reindexing some part again or... > > Thanks a lot!! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > That sounds like the conversationsdb issue I was talking about. Have you tried these steps? > > ctl_conversationsdb -z USER > ctl_conversationsdb -b USER > > I have been testing Xapian searches. Have seen, it's not able to find > strings inside the body of the email. If I set in imap.conf > "search_fuzzy_always: 1" no content is displayed in the searches of a > Roundcube stock webmail. If I remove that config value from imap.conf > and restart services, then search results appear. Does Xapian not index > the body of emails?. Does Xapian, just index the headers?. But this > affirmation does not seem to be possible in my case too... as I have in > the config "search_index_headers: no". > > I'm using the following config : > > conversations: 1 > search_engine: xapian > search_index_headers: no > search_batchsize: 8192 > defaultsearchtier: t1 > t1searchpartition-default: /expert/search > t1searchpartition-expert2: /expert2/search > t1searchpartition-expert3: /expert3/search > > Could anyone help me mates?. > > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578 Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hagedorn at uni-koeln.de Mon Jan 7 09:57:39 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 15:57:39 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Hi, --On 7. Januar 2019 um 14:51:02 +0100 Egoitz Aurrekoetxea wrote: > This seems to take ages... why don't you run it for a single account first, to make sure that it actually helps? >I'm trying to figure the best way of > implementing this and of clarifying concepts.... I'm running Squatter in > rolling replication mode and exist the concept of conversations then. > What is the exact role of each of them?. Squatter seems to index the > mailbox but when something is not properly indexed instead of running > Squatter you use ctl_conversations for reindexing some part again or... squatter is nowadays a bit of a misnomer, because it uses whatever index you have configured. In cyrus 2.4, squatter would always create a SQUAT index. When you run squatter with Xapian, it will build the index, but for the index to actually work, you also need the conversationsdb. squatter does not touch the conversationsdb! The index is only a pointer to the conversationsdb, not to actual messages. Robert can probably explain this much better than me, but I think the problem is the following: ? when you have conversations enabled in imapd.conf, normal deliveries to the mailboxes (e.g. using lmtpd) will update the conversationsdb ? syncing (at least using the "old" mechanism, not sure about sync between instances running 3.0) does *not* update the conversationsdb Once you have a running 3.0 server, you probably won't need to run ctl_conversationsdb ever again. But when you are at the stage of syncing mail from 2.4 to 3.0, you *will* need to rebuild each user's conversationdb at least once, after you have finished with syncing that user. Again, this is all based on my understanding and not an official answer. > El 07-01-2019 10:19, Sebastian Hagedorn escribi?: > >> That sounds like the conversationsdb issue I was talking about. Have you >> tried these steps? >> >> ctl_conversationsdb -z USER >> ctl_conversationsdb -b USER Mit freundlichen Gr??en Sebastian Hagedorn -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From Hagedorn at uni-koeln.de Mon Jan 7 10:01:20 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 16:01:20 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> Message-ID: <9EC01839DEEE8F7A28BAFF0D@tyrion.rrz.uni-koeln.de> That's a really good question. I had been hoping that the performance of a Xapian search would be much better than a SQUAT search, but now I'm not so sure. --On 7. Januar 2019 um 15:05:52 +0100 Egoitz Aurrekoetxea wrote: > And, by the way.... > > when using Squatter instead of Xapian as a search engine.... what do we > really lost?. Just the fact of having a statistical worse results?. Is > it Xapian faster than squat engine?. -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From Hagedorn at uni-koeln.de Mon Jan 7 10:03:16 2019 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Mon, 07 Jan 2019 16:03:16 +0100 Subject: Xapian searches of the body of an email In-Reply-To: <776585952e420b671ed40ed7495e88db@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> <776585952e420b671ed40ed7495e88db@sarenet.es> Message-ID: <94BFD098ADE0B30BCCC8CBFF@tyrion.rrz.uni-koeln.de> As I wrote before, this should only be necessary as long as you are in the syncing stage of the migration. Once all new mail is delivered to the 3.0 server everything should just work ? I think ;-) --On 7. Januar 2019 um 15:41:59 +0100 Egoitz Aurrekoetxea wrote: > ctl_conversationsdb -z and -b (with -r I assume for all users) should be > run only on new user accounts or... periodically for any user account?. > Or does squatter maintain too the conversations database?. -- .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available URL: From elias at hi.is Mon Jan 7 10:13:54 2019 From: elias at hi.is (=?utf-8?B?RWzDrWFzIEhhbGxkw7NyIMOBZ8O6c3Rzc29u?=) Date: Mon, 7 Jan 2019 15:13:54 +0000 Subject: Xapian searches of the body of an email In-Reply-To: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> Message-ID: Regarding indexing and searching in body of emails; what if the body text is encoded in base64 or quoted-printable? It won't yield any unencoded search strings, or what? -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Mon Jan 7 10:20:05 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 07 Jan 2019 16:20:05 +0100 Subject: Xapian searches of the body of an email In-Reply-To: References: <779e844a999aa3cf7c6f10d4f2a8ce62@sarenet.es> <959e8cc6ce122d625ee567a5b848c32f@sarenet.es> <776585952e420b671ed40ed7495e88db@sarenet.es> Message-ID: <1e32afef563dc163514211cd98ddb13f@sarenet.es> Hi Nic, Sorry for the noise. Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 07-01-2019 16:00, Nic Bernstein escribi?: > Could you please stop cross-posting to Cyrus-Info and Cyrus-Devel? I can understand if, in your first message, you're not sure which list to post on, but to send copies of an entire thread to both lists in tiresome. There are many people who subscribe to both lists. > > Thank you in advance, > -nic > > On 1/7/19 8:41 AM, Egoitz Aurrekoetxea wrote: > >> Sorry for asking again.. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Fri Jan 25 08:32:34 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Fri, 25 Jan 2019 13:32:34 +0000 Subject: Generating MIME boundaries within cyrus Message-ID: Hello, sending a MIME message implies generating multipart boundary (random) string. Cyrus does this in: imap/cyrdump.c:g enerate_boundary() imap/httpd.c:write_multipart_body() imap/http_caldav_sched.c:imip_send_sendmail() imap/jmap_mail.c:_mime _make_boundary() imap/lmtp_siele.c:send_rejection() Isn?t it feasible to avoid on this occassion code duplication? Regards ????? From brong at fastmailteam.com Fri Jan 25 20:29:05 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Fri, 25 Jan 2019 20:29:05 -0500 Subject: Generating MIME boundaries within cyrus In-Reply-To: References: Message-ID: <848f5983-f52c-46b0-8750-ce460dd7970e@beta.fastmail.com> Yes, a general "create mine boundary" function could be written. I guess nobody figured it was worth the effort yet. Bron. On Sat, Jan 26, 2019, at 00:33, ????? ???????? wrote: > Hello, > > sending a MIME message implies generating multipart boundary (random) string. Cyrus does this in: > > imap/cyrdump.c:g > enerate_boundary() > imap/httpd.c:write_multipart_body() > imap/http_caldav_sched.c:imip_send_sendmail() > imap/jmap_mail.c:_mime > _make_boundary() > imap/lmtp_siele.c:send_rejection() > > Isn?t it feasible to avoid on this occassion code duplication? > > Regards > > ????? > > -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Mon Jan 28 17:45:41 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Mon, 28 Jan 2019 17:45:41 -0500 Subject: Notes 29/1/2019 Message-ID: <55447381-7b60-4d78-8535-8d1c332e95bc@www.fastmail.com> Present: Ken, Bron, Robert, Partha, ellie Ken: * was in Philly last week, so did very little Cyrus programming * found some bugs in scheduling code Robert: * Had a short week last week * Working on rewriting cyrus.indexed.db for Xapian to use mailbox uuids rather than names - Bron to review * Tested locally and on fastmailtest. * When cyrus.indexed.db is read, is migrated on the fly. - will test to see if the indexed.db is read for search, or only in squatter. * Had to update a cassandane test built on internal format of indexed.db * tested revert on fastmailtest.com * to finish, needs UUID changes to conversations.db to land. * will land it along with Ken's storage by UUID patches. * also: updated JMAP Mailbox/set to update subscriptions DB. * working on Xapian multilingual feature - hope to have a demo this week. - do we want to support i;unicode-casemap ? - suspect we'll discover that no clients use this * either way, may need to change uni2searchform to support multilingual correctly. * also the snippet generator needs to know which language was matched to correctly identify snippets. * maybe persist detected languages as an annotation? Rik: * will have a look at the i;unicode-casemap algorithm. * hit calendar-address-set bugs Bron: * fixed ACL creation bug in children of INTERMEDIATE folders * fixed readonly conversations DB issue with XCONV commands because expunge wants to write ellie: * ongoing hosting and relicensing discussion * been doing mailing list discussions * updated build environment to use cyruslibs and now testing against zeroskip Cyrus needs a way to translate from username to identities. * a lookup that does userid -> email address or email address -> "who does this belong to". * is this user local on this server? - for free-busy queries, need a different lookup that returns "is this user in the same customer as me". * ptloader and ldap support for group RACLs already works * maybe implement a redis protocol cyrusdb backend and redis protocol server in postfixlookup at FastMail or whatever Next week Robert, Bron and Ken will be in Zurich, so we won't be having an online meeting next week. -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: