Xapian index not being used for message search in Roundcube

[ellie timoney]

I don't really understand search in any depth, but it's interesting to observe that, in addition to the different command (SEARCH vs THREAD), those two searches are also using different search criteria ("BODY linux" vs "TEXT linux").

It might be informative to try do the SEARCH search with "TEXT linux" instead of "BODY linux", to narrow down whether the difference is due to the use of the SEARCH vs THREAD  command, or the use of the "BODY" vs "TEXT" search key?

Looking at the source on master, SEARCH and THREAD both seem to be using the same search API, so at a glance it seems like they should both be using Xapian if either is.  And looking at the commit dates on those functions, it doesn't look like it's changed substantially since 3.0, at least not at a level I can easily see.

I had a quick look at the RFC, and "BODY" searches just the message body, whereas "TEXT" searches both body and headers.  So I wonder if the difference is that TEXT needs to open all the message files to read the headers, whereas BODY can just return results straight from the Xapian index?

I'm not sure if there's been changes to header searching (like, maybe we index more of the header content?) since 3.0, but this is getting beyond what I know off the cuff or can just casually look up.

Anyway, if you could try "UID SEARCH TEXT linux" and see if that's similarly slow to the THREAD version, that would give us a definite pointer in the right direction.

Cheers,

ellie

On Sun, Feb 23, 2020, at 10:11 PM, Frederik Himpe via Info-cyrus wrote:
> I have configured Cyrus 3.0.13 with the Xapian search engine and
> enabled search_fuzzy_always. This appears to work fine when I search in
> the message body using the Evolution mail client, as I get a response
> quickly:
> 
> <1582453709<L03163 UID SEARCH BODY linux
> >1582453713>* SEARCH 226927
> 226929 226964 226974 226999 227215 227238 [...]
> L03163 OK Completed (643
> msgs in 0.970 secs)
> 
> However when I search messages using the Roundcube webmail client,
> Roundcube does not get a response in time and shows no results. An
> strace of the imapd proceess indicates it is STATing, OPENing and
> MMAPing all files in the mailbox.
> 
> This is the log:
> <1582455581<A0004 UID THREAD REFS US-ASCII ALL UNSEEN TEXT Linux
> >1582455723>* THREAD
> (229566)(229570)(229574)(229599)(229618)(229639)[...]
> A0004 OK Completed (157 msgs in 11.340 secs)
> 
> So it appears Roundcube is using a different command to search. Is it
> expected that this command does not use the Xapian search engine? Is
> there a way to make it use it?
> 
> Some relevant snippets from imapd.conf:
> sync_log: on
> sync_log_channels: squatter
> 
> conversations: 1
> search_engine: xapian
> search_index_headers: no
> search_batchsize: 8192
> search_fuzzy_always: 1
> defaultsearchtier: temp
> tempsearchpartition-default: /var/lib/cyrus/search.temp
> datasearchpartition-default: /var/lib/cyrus/search.data
> 
> cyrus.conf:
> 
> EVENTS {
>         squatter1       cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus 
> squatter -z data -t temp,data" at=0517
> 
> }
> DAEMON {
>   squatter cmd="squatter -R"
> }
> 
> 
> Regards,
> 
> -- 
> Frederik Himpe <frederik at frehi.be>
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>