prblems rebuilding conversations db

[Michael Menge]

Hi

Quoting ellie timoney <ellie at fastmail.com>:
...
> On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote:
...
>> Quoting ellie timoney <ellie at fastmail.com>:
...
>> > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
>> >> Hi,
>> >>
>> >> because conversations db seems to be required for search indexes, I
>> >> enabled this option
>> >> on our production servers today and tried to rebuild the conversations
>> >> db for all users with
>> >>
>> >> ctl_conversationsdb -v -b UID
>> >>
>> >> For most users this did take less than a second. But for some users
>> >> this process would
>> >> not finish. I did kill one process after about 30 Minutes (most others
>> >> after 3 Minutes).
>> >> The UserID.conversations has grown over 2 GB (the mailbox itself has
>> >> only ~700 MB of mails,
>> >> and the conversations files from finnished rebuilds are less then 20 MB)
...
>> I did run ctl_conversationsd -b with strace and discovered that the inbox
>> (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache  cyrus.O_RDWR )  
>> of that user
>> was opened over and over again, no other folder for that user was accessed.
>> For other users the inbox was only opened once and then the other
>> folders followed.
>> So not stopping "ctl_conversationsd -b", it would have run till my
>> filesystem was
>> full.
>>
>
> I guess something about that inbox is confusing it, and making it  
> redo it over and over?  Very curious
>
...
>> > Though, that raises the question: was the user accessing the mailbox
>> > while the rebuild was in progress?  I wonder if their client is
>> > doing something funny and tripping things up?
>>
>> We did this while the server was accessible (didn't want to have
>> several hour downtime for ~44000 users)
>
> Fair enough!
>
>> But as I did my recent testing on a test-server without user access,
>> only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during
>> the conversation rebuild mentioned in this mail. So I don't think that
>> user access is part of the problem.
>
> Thanks for testing that, good to rule it out as a culprit
>
>> So for me it looks like we have two problems:
>>
>> 1. multiple entries for the same key in skiplist db files.
>>     As skiplist is a Key/Value store this should not be possible
>>     to have duplicate keys?
>
> I don't know the internals of our database formats well, but I  
> _think_ maybe it writes new key/value pairs by appending to the end  
> of the database, and then updating the skiplist structure to point  
> to the newer version instead of the older version.  If this is  
> correct, then if it's rewriting the keys over and over for some  
> reason, I would expect the raw file to contain a lot of old,  
> unlinked records for the same key, but which aren't visible over the  
> API.  I believe there's a process that runs occasionally that  
> rewrites these databases with only the "live" keys and gets back the  
> wasted space?
>

I did discover, because of the loop (see below) the point of
database commit was not reached, so I guess that could explain
that there was no duplicated key cleanup done.

>>     Is twoskip a better alternative?
>
> This is before my time, but twoskip was written to solve problems  
> FastMail experienced with using skiplist with conversations  
> ("post-crash recovery was too slow").  So if there's a pathological  
> case in the more recent conversations code that trips a bug in  
> skiplist, we probably wouldn't see it.  So, I'd recommend using  
> twoskip for conversationsdb just based on that :)

Same problem with twoskip, but as I said above, the commit point
was not reached.

>
>> 2. Why didn't ctl_conversationsdb continue to process
>>     the next mailbox but re-did the INBOX / the last mailbox
>
> It kinda sounds to me like it considered the operation to be  
> unsuccessful, and tried again?  Not sure what would cause this.  Is  
> there anything interesting in syslog?
>
>> I will try to run a debugger on ctl_conversationsdb and
>> will test with twoskip as conversations db format
>

The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)

For the problematic mailbox that I tested, for every message
record->cid was NULLCONVERSATION, so mailbox_cacherecord,
message_update_conversations and mailbox_rewrite_index_record
where called, each returned 0.

After iterating trough all messages in the mailbox count was > 0, and r==0,
so the while condition (!r && count) was true for the next run.
At this point record->cid was still NULLCONVERSATION for every message,
which I guess should not be the case.

> Thanks, it'd be very interesting to see if this issue reproduces  
> with the twoskip format!
>

Yes, it did.

>> I did notice one other problem, recreating the conversations db
>> on the replica confused the sync protocol as the rebuild did increase
>> the modseq.
>
> Do you mean to say that this user's conversations db could be  
> rebuilt successfully on the replica?  Or was this with one of the  
> "good" users?
>

I did activated the conversation db first on the replica (to watch the  
performance
impact and time needed) and did the rebuild with the -r flag.

I didn't notice an endless loop on the replic, but remember the creation of
the conversation db failed for some users and I had no good way to know for
which user the process was successful and for which it failed I decided to
run a single process for each user and check the return value.

I will test if the -r flag changes the rebuild on my test-server.

...

--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen