squatter not used after upgrade to cyrus 3.0.8

Fri May 10 07:30:36 EDT 2019

Hi Michael,

Do you resolved this problem?
I'm having the same issue with cyrus 3.0.9 not accessing cyrus.squat files.

I've put in my impad.conf:
   search_engine: squat
   search_fuzzy_always: 1

Any recomendation will be appreciated.

Best regards,
Carlos.

El 25/10/2018 a las 15:09, Michael Menge escribió:
> Hi,
>
> Quoting Michael Menge <michael.menge at zdv.uni-tuebingen.de>:
>
>> Hi,
>>
>> Quoting Albert Shih <Albert.Shih at obspm.fr>:
>>
>>> Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit
>>>> Hi,
>>>>
>>>> we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
>>>> problems
>>>> which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
>>>> problem
>>>> we receive reports about is, that the search is not working/too slow.
>>>>
>>>> First we discovered that one of the options for Squatter are not supported
>>>> anymore, "-s Skip mailboxes whose index file is older than their current
>>>> squat file (within a small time delta)." and that squatter does not like
>>>> "-r" in combination with the source "user"
>>>>
>>>>   > squatter -C /etc/imapd_be.conf -r  user
>>>>   fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
>>>> 2339: keylen
>>>>
>>>>
>>>> But after reindexing all mailboxes the search is still slow. I tried to
>>>> debug this and
>>>> found with strace that cyrus didn't try to open the cyrus.squat file for the
>>>> mailbox.
>>>>
>>>> I suspect that I am missed some configuration change. So here is our
>>>> imapd.conf for our backends
>>>
>>> If I'm correct you need :
>>>
>>>  search_fuzzy_always: on
>>>
>>> in your config.
>>>
>>> If you can tell me if it's work...I would really appreciate. Because I
>>> activated that but I'm not able to see if it's work really.
>>>
>> Thanks for your help.
>>
>> I did tried it on my test server and it seems to be a bit faster,
>> but that could be due to caching. Strace still didn't show any access
>> to the cyrus.squat file.
>>
>> For information: We only use squatter. No Xapia. And we had much faster
>> search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
>> the old system but our users are complaining and they receive timeouts
>> in our horde/imp webmailer, which they did't receive before.
>>
>> Any other ideas are appreciated.
>
> I still have the problem that search in cyrus imap 3.0.8 with search engine
> squatter is slow compared to 2.4.20. I try to figure out if the squatter
> search engine is working in cyurs imapd 3.0 and I messed up my configuration,
> or if my configuration should work but squatter is broken.
>
> I did set up a test environment to compare the old and new versions.
> To verify that the search is indeed slower with 3.0.8
>
> I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH HEADER 
> X-comment Unirundmail'
>
> === 2.4.20 === SEARCH TEXT
>
> A SELECT INBOX
> * 4321 EXISTS
> * 4321 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
> * OK [UNSEEN 1] Ok
> * OK [UIDVALIDITY 1540372444] Ok
> * OK [UIDNEXT 93369] Ok
> * OK [HIGHESTMODSEQ 2] Ok
> * OK [URLMECH INTERNAL] Ok
> A OK [READ-WRITE] Completed
> B SEARCH TEXT "Test"
> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
> 27 28 29 30 31 33 34 35 36 37 38 39
> ....
> 4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 
> 4317 4318 4321
> B OK Completed (1996 msgs in 0.690 secs)
>
>
> === 3.0.8 === SEARCH TEXT
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
> $label1 $label2 $label3 hordetest testflag)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok
> * OK [UNSEEN 3737] Ok
> * OK [UIDVALIDITY 1238498084] Ok
> * OK [UIDNEXT 93373] Ok
> * OK [HIGHESTMODSEQ 98491] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a OK [READ-WRITE] Completed
> B SEARCH TEXT "Test"
> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
> 27 28 29 30 31 33 34 35 36 37 38 39
> ....
> 4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295 4296 4298 
> 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317 4318 4321
> B OK Completed (1935 msgs in 2.580 secs)
>
> ==== 2.4.20 === SEARCH HEADER
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
> * OK [UNSEEN 1] Ok
> * OK [UIDVALIDITY 1540372444] Ok
> * OK [UIDNEXT 93369] Ok
> * OK [HIGHESTMODSEQ 2] Ok
> * OK [URLMECH INTERNAL] Ok
> a OK [READ-WRITE] Completed
> b SEARCH HEADER X-comment Unirundmail
> * SEARCH 4283 4291 4313 4319 4320
> b OK Completed (5 msgs in 0.020 secs)
>
> ==== 3.0.8 === SEARCH HEADER
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
> $label1 $label2 $label3 hordetest testflag)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok
> * OK [UNSEEN 3737] Ok
> * OK [UIDVALIDITY 1238498084] Ok
> * OK [UIDNEXT 93373] Ok
> * OK [HIGHESTMODSEQ 98491] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a OK [READ-WRITE] Completed
> b SEARCH HEADER X-comment Unirundmail
> * SEARCH 4283 4291 4313 4319 4320
> b OK Completed (5 msgs in 0.370 secs)
>
> ===
>
> There is also a big discrepancy between time indicated in the "OK Completed" 
> and the time from
> sending the search command till the return of the result, which is 0.890 sec 
> compared to ~30 sec
> on the production system.
>
> I used strace on the imapd processes while searching to verify that the squat 
> file was used
> in 2.4 but not in 3.0.
> I could see open events for the squat file and the messages that where found 
> for 2.4.20
> and no open event (not even a failed one) to the squat file but instead open 
> events for
> all message files in that folder for 3.0.8
>
> I read the documentation and source code and tried to figure out if i am 
> missing some
> config options, or if i could pinpoint a function where the search was turning 
> the
> wrong way. I used "perf -g" the see the call graphs and to figure out where the
> call graphs change
>
> I can see that the same functions are called up to "index_search", and that 
> the called functions
> change at that point. I know that the search code got restructured to support 
> different search
> engines and that some functions got renamed. I have attached the perf report 
> output, so that
> someone with a better understanding of the code can see whats going on. I can 
> provide the
> perf.data files if it helps.
>
> Can someone confirm or refute that the squatter search engine is working with 
> cyrus imapd 3.0.x?
>
> Is "search_engine: squat" in imapd.conf and a "squatter" event in cyrus.conf 
> is sufficient to
> use the squatter search index in 3.0 or are there other options steps required.
>
> Regards
>
>    Michael Menge
>
> PS. link to my original post with my imapd.conf
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-September/040395.html
>
>
>
>
>
>
>
> --------------------------------------------------------------------------------
> M.Menge                                Tel.: (49) 7071/29-70316
> Universität Tübingen                   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung          mail: michael.menge at zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/ec8eecd6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3627 bytes
Desc: Firma criptogr??fica S/MIME
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/ec8eecd6/attachment-0001.p7s>