Re: thoughts about conversation db and cyrus [faild to rename deleted mailbox, or how conversation db sucks again]

Fri Mar 8 06:45:04 EST 2019

Thanks Michael - I don't have much time to give a longer response right now, but this is really valuable feedback into what we focus on over the next few months and into the 3.2 release plans. I really appreciate you taking the time to spell this out in detail.

Cheers,

Bron.

On Tue, Mar 5, 2019, at 02:17, Michael Menge wrote:
> Hi Ellie and Bron,
> 
> first thank you for your ideas for the workaround and for opening the issue.
> 
> I have to apologize about the original subject, but I was a bit 
> frustrated that
> I have encountered again a problem with the conversation db.
> 
> As I have discovered jet another, not jet reported problem with 
> conversation db
> (I am unsure if it's a problem with the conversation db or if only 
> shows an other
> problem some where else), I have decided to deactivate the conversation db for
> the moment.
> 
> I think I should elaborate the general problem I have with this 
> feature and the
> cyrus development as i have observed it.
> 
> I am an experienced linux and cyrus administrator. I am not an 
> software developer,
> or programmer, but I understand enough about programming to fix small problems
> and narrow down problems.
> 
> Cyrus code quality has grown in the last 14 years since i first set up 
> our first
> cyrus imap (2.3) server. Especially since Fastmail dedicated personal for the
> cyrus project. But also automatic testing and other design decisions helped to
> bring the project a big step forward.
> 
> I know we have a complex setup (murder, replication, meta- and archive 
> partition,
> delayed delete, delayed expunge) so that we use combinations of 
> features that are not
> that common. So fare we are happy with our cyrus setup.
> 
> We did not encounter any data loss, and where able to fix most 
> problems in short time.
> The system is very stable and expect for the slow search can't 
> complain about the performance.
> So thanks again to the devs for the work and also for the community 
> the help I received
> in the last 14 years.
> 
> I like new features, but I have always to balance the advantages of 
> these new features
> with the impact on stability, performance and administration overhead. 
> In that regard I
> tend to exercise caution.
> 
> While testing cyrus 3.0, on one hand neither conversation db nor 
> sphinx or xapian search
> engines looked particular interesting, as I had no need to improve 
> search speed in
> cyrus 2.4 with squatter ("If it isn't broken don't fix it"). On the 
> other hand i didn't
> have time to rigorous test these features, so I decided not to use 
> them as new feature
> have a tendency to contain more new bugs.
> 
> After upgrading the production system to cyrus 3.0 I discovered that 
> search was slow with
> squatter, conversation db did more than the documentation suggested 
> that it does.
> I don't know if it was intended that conversation db was required for 
> squatter to work
> or if it broke by one change to support multiple search engines but 
> the requirement
> was surprising. I did try to find the commit that did break the 
> squatter search but
> failed, as was unable to compile most commits git bisect suggested.
> 
> The problems I encountered with conversation db, confirmed my initial 
> caution not to enable
> conversation db without testing. But it would be unfair to blame only 
> the implementation of
> conversation db.
> 
> One problem was caused by "reconstruct -V max" not upgrading all 
> mailboxes, which i did
> miss in the logs. It would be nice if ctl_conversationsdb would check 
> mailbox version
> before creating a huge conversation db file in an endless loop.
> 
> The problem with "IOERROR: conversations_audit on load:" and "IOERROR: 
> conversations_audit on store:"
> is still a mystery to so it is unclear if it is really a bug in the 
> conversation db of if it shows
> an other problem with my installation.
> 
> The same is with the new problem i had no time to report jet. Deleting 
> users i see the following error
> popping up for some accounts.
> "Fatal error: Internal error: assertion failed: imap/conversations.c: 
> 2205: !status.exists"
> 
> But breaking an common use case of an other feature like "delete_mode: 
> delayed" is
> an other case. This should have been fixed before it was released in 
> the stable
> cyrus version. At least a WARNING in the documentation is required.
> 
> I would like to help to improve the documentation, but there are some 
> questions that need to be answered:
> 
> 1. Which search engines and combinations are currently supported?
>  Is a stand alone squatter still supported?
> 
> 2. What are to pros and cons for the supported search engines?
> 
> 3. Should, or to which extend should, the search engines work without 
> conversations db?
>  Or is enabling the conversations db a new requirement for some/all 
> search engines?
> 
> 4. Is the conversations db murder aware? And how do shared folders 
> (one user shared one
>  of his folders with other users) on the same server/cross server
>  affect search results and performances
> 
> 5. What is stored in the conversations db?
> 
> https://www.cyrusimap.org/dev/imap/concepts/deployment/databases.html#conversations-userid-conversations
>  is incomplete as conversations db also contains hashes of mime parts.
> 
> 6. Which Information is affected by conversations_expire_days
> 
> TLDR; I like cyrus, but there is some work to do regarding to 
> conversation db and search engines,
> in the field of documentation, code testing and feature interaction
> 
> 
> --------------------------------------------------------------------------------
> M.Menge Tel.: (49) 7071/29-70316
> Universität Tübingen Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung mail: 
> michael.menge at zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190308/00218a98/attachment.html>