prblems rebuilding conversations db
michael.menge at zdv.uni-tuebingen.de
Thu Jan 24 05:12:37 EST 2019
Quoting ellie timoney <ellie at fastmail.com>:
> On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote:
>> Quoting ellie timoney <ellie at fastmail.com>:
>> > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
>> >> Hi,
>> >> because conversations db seems to be required for search indexes, I
>> >> enabled this option
>> >> on our production servers today and tried to rebuild the conversations
>> >> db for all users with
>> >> ctl_conversationsdb -v -b UID
>> >> For most users this did take less than a second. But for some users
>> >> this process would
>> >> not finish. I did kill one process after about 30 Minutes (most others
>> >> after 3 Minutes).
>> >> The UserID.conversations has grown over 2 GB (the mailbox itself has
>> >> only ~700 MB of mails,
>> >> and the conversations files from finnished rebuilds are less then 20 MB)
>> I did run ctl_conversationsd -b with strace and discovered that the inbox
>> (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache cyrus.O_RDWR )
>> of that user
>> was opened over and over again, no other folder for that user was accessed.
>> For other users the inbox was only opened once and then the other
>> folders followed.
>> So not stopping "ctl_conversationsd -b", it would have run till my
>> filesystem was
> I guess something about that inbox is confusing it, and making it
> redo it over and over? Very curious
>> > Though, that raises the question: was the user accessing the mailbox
>> > while the rebuild was in progress? I wonder if their client is
>> > doing something funny and tripping things up?
>> We did this while the server was accessible (didn't want to have
>> several hour downtime for ~44000 users)
> Fair enough!
>> But as I did my recent testing on a test-server without user access,
>> only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during
>> the conversation rebuild mentioned in this mail. So I don't think that
>> user access is part of the problem.
> Thanks for testing that, good to rule it out as a culprit
>> So for me it looks like we have two problems:
>> 1. multiple entries for the same key in skiplist db files.
>> As skiplist is a Key/Value store this should not be possible
>> to have duplicate keys?
> I don't know the internals of our database formats well, but I
> _think_ maybe it writes new key/value pairs by appending to the end
> of the database, and then updating the skiplist structure to point
> to the newer version instead of the older version. If this is
> correct, then if it's rewriting the keys over and over for some
> reason, I would expect the raw file to contain a lot of old,
> unlinked records for the same key, but which aren't visible over the
> API. I believe there's a process that runs occasionally that
> rewrites these databases with only the "live" keys and gets back the
> wasted space?
I did discover, because of the loop (see below) the point of
database commit was not reached, so I guess that could explain
that there was no duplicated key cleanup done.
>> Is twoskip a better alternative?
> This is before my time, but twoskip was written to solve problems
> FastMail experienced with using skiplist with conversations
> ("post-crash recovery was too slow"). So if there's a pathological
> case in the more recent conversations code that trips a bug in
> skiplist, we probably wouldn't see it. So, I'd recommend using
> twoskip for conversationsdb just based on that :)
Same problem with twoskip, but as I said above, the commit point
was not reached.
>> 2. Why didn't ctl_conversationsdb continue to process
>> the next mailbox but re-did the INBOX / the last mailbox
> It kinda sounds to me like it considered the operation to be
> unsuccessful, and tried again? Not sure what would cause this. Is
> there anything interesting in syslog?
>> I will try to run a debugger on ctl_conversationsdb and
>> will test with twoskip as conversations db format
The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)
For the problematic mailbox that I tested, for every message
record->cid was NULLCONVERSATION, so mailbox_cacherecord,
message_update_conversations and mailbox_rewrite_index_record
where called, each returned 0.
After iterating trough all messages in the mailbox count was > 0, and r==0,
so the while condition (!r && count) was true for the next run.
At this point record->cid was still NULLCONVERSATION for every message,
which I guess should not be the case.
> Thanks, it'd be very interesting to see if this issue reproduces
> with the twoskip format!
Yes, it did.
>> I did notice one other problem, recreating the conversations db
>> on the replica confused the sync protocol as the rebuild did increase
>> the modseq.
> Do you mean to say that this user's conversations db could be
> rebuilt successfully on the replica? Or was this with one of the
> "good" users?
I did activated the conversation db first on the replica (to watch the
impact and time needed) and did the rebuild with the -r flag.
I didn't notice an endless loop on the replic, but remember the creation of
the conversation db failed for some users and I had no good way to know for
which user the process was successful and for which it failed I decided to
run a single process for each user and check the return value.
I will test if the -r flag changes the rebuild on my test-server.
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.menge at zdv.uni-tuebingen.de
More information about the Info-cyrus