thoughts about conversation db and cyrus [faild to rename deleted mailbox, or how conversation db sucks again]

[Michael Menge]

Hi Ellie and Bron,

first thank you for your ideas for the workaround and for opening the issue.

I have to apologize about the original subject, but I was a bit  
frustrated that
I have encountered again a problem with the conversation db.

As I have discovered jet another, not jet reported problem with  
conversation db
(I am unsure if it's a problem with the conversation db or if only  
shows an other
problem some where else), I have decided to deactivate the conversation db for
the moment.

I think I should elaborate the general problem I have with this  
feature and the
cyrus development as i have observed it.

I am an experienced linux and cyrus administrator. I am not an  
software developer,
or programmer, but I understand enough about programming to fix small problems
and narrow down problems.

Cyrus code quality has grown in the last 14 years since i first set up  
our first
cyrus imap (2.3) server. Especially since Fastmail dedicated personal for the
cyrus project. But also automatic testing and other design decisions helped to
bring the project a big step forward.

I know we have a complex setup (murder, replication, meta- and archive  
partition,
delayed delete, delayed expunge) so that we use combinations of  
features that are not
that common. So fare we are happy with our cyrus setup.

We did not encounter any data loss, and where able to fix most  
problems in short time.
The system is very stable and expect for the slow search can't  
complain about the performance.
So thanks again to the devs for the work and also for the community  
the help I received
in the last 14 years.

I like new features, but I have always to balance the advantages of  
these new features
with the impact on stability, performance and administration overhead.  
In that regard I
tend to exercise caution.

While testing cyrus 3.0, on one hand neither conversation db nor  
sphinx or xapian search
engines looked particular interesting, as I had no need to improve  
search speed in
cyrus 2.4 with squatter ("If it isn't broken don't fix it").  On the  
other hand i didn't
have time to rigorous test these features, so I decided not to use  
them as new feature
have a tendency to contain more new bugs.

After upgrading the production system to cyrus 3.0 I discovered that  
search was slow with
squatter, conversation db did more than the documentation suggested  
that it does.
I don't know if it was intended that conversation db was required for  
squatter to work
or if it broke by one change to support multiple search engines but  
the requirement
was surprising. I did try to find the commit that did break the  
squatter search but
failed, as was unable to compile most commits git bisect suggested.

The problems I encountered with conversation db, confirmed my initial  
caution not to enable
conversation db without testing. But it would be unfair to blame only  
the implementation of
conversation db.

One problem was caused by "reconstruct -V max" not upgrading all  
mailboxes, which i did
miss in the logs. It would be nice if ctl_conversationsdb would check  
mailbox version
before creating a huge conversation db file in an endless loop.

The problem with "IOERROR: conversations_audit on load:" and "IOERROR:  
conversations_audit on store:"
is still a mystery to so it is unclear if it is really a bug in the  
conversation db of if it shows
an other problem with my installation.

The same is with the new problem i had no time to report jet. Deleting  
users i see the following error
popping up for some accounts.
"Fatal error: Internal error: assertion failed: imap/conversations.c:  
2205: !status.exists"

But breaking an common use case of an other feature like "delete_mode:  
delayed" is
an other case. This should have been fixed before it was released in  
the stable
cyrus version. At least a WARNING in the documentation is required.

I would like to help to improve the documentation, but there are some  
questions that need to be answered:

1. Which search engines and combinations are currently supported?
    Is a stand alone squatter still supported?

2. What are to pros and cons for the supported search engines?

3. Should, or to which extend should, the search engines work without  
conversations db?
    Or is enabling the conversations db a new requirement for some/all  
search engines?

4. Is the conversations db murder aware? And how do shared folders  
(one user shared one
     of his folders with other users) on the same server/cross server
     affect search results and performances

5. What is stored in the conversations db?
     
https://www.cyrusimap.org/dev/imap/concepts/deployment/databases.html#conversations-userid-conversations
    is incomplete as conversations db also contains hashes of mime parts.

6. Which Information is affected by conversations_expire_days

TLDR; I like cyrus, but there is some work to do regarding to  
conversation db and search engines,
in the field of documentation, code testing and feature interaction


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen