prblems rebuilding conversations db

[ellie timoney]

Hi Michael,

On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
> Hi,
> 
> because conversations db seems to be required for search indexes, I  
> enabled this option
> on our production servers today and tried to rebuild the conversations  
> db for all users with
> 
> ctl_conversationsdb -v -b UID
> 
> For most users this did take less than a second. But for some users  
> this process would
> not finish. I did kill one process after about 30 Minutes (most others  
> after 3 Minutes).
> The UserID.conversations has grown over 2 GB (the mailbox itself has  
> only ~700 MB of mails,
> and the conversations files from finnished rebuilds are less then 20 MB)

So, a 2GB-and-growing conversations db for a 700MB mailbox is a ratio of ~3:1, which seems obscene...

For comparison, the Conversations.append_reply_200 cassandane test (two messages with 100 replies each) produces an 87Kb conversations db for a 1016Kb mailbox (ratio ~1:12, n.b inverted!), which makes your ~3:1 and growing ratio seem even more obscene...

Obviously you can't share the bad conversations db because conversations db's are full of personally-identifying information.  But I wonder if looking through it with `cyr_dbtool ... show` turns up any unusual patterns, especially in comparison to one of the good users?  I wonder if it's somehow repeating itself?

Bron, does this sound like anything you've seen before, that maybe got fixed on master but not backported to 3.0?
 
> Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock  
> user.UserID for 1.5 seconds"
> every few seconds.

This message is logged when the lock is released, if that lock had been held for >1s.  It's reporting that it held the lock for longer than it would like to have, but at least you know the lock has been released.  I think this is telling you that user-visible performance on the mailbox would have been impacted while this was occurring (because imapd/lmtpd wouldn't be able to access their mailbox while ctl_conversationsdb was holding those locks), but also tells you that ctl_conversationsdb was doing the right thing, and not just holding a single lock for the entire job.

Though, that raises the question: was the user accessing the mailbox while the rebuild was in progress?  I wonder if their client is doing something funny and tripping things up?

Cheers,

ellie

> I did try reconstruct -r -G user/UserID, followed by   
> ctl_conversationsdb -v -z UID
> and tried to rebuild the db again. Reconstruct reported no errors, and  
> the new rebuild
> hat the same problem.
> 
> Has anybody seen a similar problem?
> 
> Any ideas how to fix this?
> 
> 
> We are running cyrus-imapd 3.0.8
> 
> 
> Kind regards
> 
>     Michael Menge
> 
> --------------------------------------------------------------------------------
> M.Menge                                Tel.: (49) 7071/29-70316
> Universität Tübingen                   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung          mail:  
> michael.menge at zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen