UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder.

[Jesper Schmitz Mouridsen]

Hi

Thanks for the debugging hints!

client_timeout sat to 30M and the UID THREAD REFS US-ASCII ALL actually completes.
But first after ~10 mins on a CPU: Intel(R) Celeron(R) CPU  N2930  @ 1.83GHz (1833.38-MHz K8-class CPU)
and after 69.484 secs on a CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (The same jail, moved to faster hardware)
63994 messages in around 40k threads.
What could cause the much spent time building the thread data? version is still 3.2.2, but 3.0.13 shows the same
behavior. The thread output of the slow folder is here FWIW https://pastebin.com/PpiJ8X2E

Regards
/Jesper


On 03.07.2020 01.52, ellie timoney wrote:
> Hi,
>
> I think I would do something like:
>
> 0) set client_timeout to a big value (see below)
> 1) let the imapd start normally
> 2) connect to it with a minimal imap client (like imtest or telnet)
> 3) check logs to see which imapd process id your client is connected to (if there's more than one)
> 4) use the "gdb /path/to/binary pid" invocation to attach gdb to the running imap process
> 5) repeat #4 in a few different ways if it's unable to find symbols
> 6) set a breakpoint on the cmd_thread function (since that's what handles the THREAD command) and then continue
> 7) back in your imap client, send the UID THREAD REFS US-ASCII ALL" command and step through to see what happens
>
> Note that these are not exhaustive steps, more of a "get started, and then react accordingly, depending on what you see"
>
> I would also try variations of that THREAD command -- if you add/remove options, does it start working?  Can you isolate the problem to a specific combination of options?  Does it only happen for some mailboxes?
>
> You probably want to set client_timeout to a big value.  The default is 10 seconds, so the server will probably throw your client off while you're reading output from gdb, and then you'll wind up debugging the "disconnect an unresponsive client code" instead.  I usually set this to be 30 minutes or so for debugging.  For 3.2 and newer, you can spell this as "30m".  For older versions, the value is in seconds, so you want "1800".
>
> Permissions may be awkward.  You will need to be the cyrus user (or root) to connect gdb to the running imapd, but you will also need permission to read the source that it was built from, which is probably not owned by the cyrus user.  In my case it's under my user account's home directory, so I add the cyrus user to my user account's group, and make sure the path to the source directory is g+rx (because I don't like running gdb as root).
>
> Cheers,
>
> ellie
>
> On Fri, Jul 3, 2020, at 5:23 AM, Jesper Schmitz Mouridsen via Info-cyrus wrote:
>> Hi.
>>
>> I recently upgraded Cyrus to 3.2.2 from 3.0.13.
>>
>> Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with
>>   >50K mails,
>>
>> makes imapd process use 100 CPU% without any progress. truss[1] or dtrace
>>
>> shows no output. The process seems totally stuck.
>>
>> I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of
>> how to use gdb to
>>
>> see what  is going one. I could not make imapd run under gdb even with
>> the -D option
>>
>> and debug_command setting or reading
>> https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug
>>
>> It is fast enough on other folders also with ~50k mails < 3 secs.
>>
>> Any hints, highly appreciated :-)
>>
>> [1] https://www.freebsd.org/cgi/man.cgi?truss
>>
>> Regards
>>
>> Jesper
>>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus