thread=refs
Ken Murchison
murch at andrew.cmu.edu
Tue Jul 19 12:00:28 EDT 2016
On 07/19/2016 11:49 AM, Gabriele Bulfon wrote:
> I found that expecially the function "_index_thread_ref" is quite
> different (arguments and implementation) in 2.4.12, while it's very
> similar in 2.5.8 : is it safe to upgrade binaries from 2.4.12 to 2.5.8
> on a running system, or do I need any kind of conversion?
doc/install-upgrade.html
>
> Also, the diff around "index_msgdata_free" where "if (!md) continue;"
> is added, is quite different from 2.5.8 too, but it's probably not
> necessary, as far as I can see...can you check this?
It is necessary. It would be necessary in 2.5.8 if this patch were to
be backported.
> ----------------------------------------------------------------------------------------
> *Sonicle S.r.l. *: http://www.sonicle.com <http://www.sonicle.com/>
> *Music: *http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
>
> ------------------------------------------------------------------------
>
>
> *Da:* Ken Murchison <murch at andrew.cmu.edu>
> *A:* gbulfon at sonicle.com info-cyrus at lists.andrew.cmu.edu
> *Data:* 19 luglio 2016 15.02.36 CEST
> *Oggetto:* Re: thread=refs
>
>
> I don't think much has changed in the threading code for a while.
> I would expect that the patch would apply pretty cleanly to 2.4.x.
>
>
> On 07/19/2016 02:58 AM, Gabriele Bulfon wrote:
>> Wow! This is really interesting!
>> What minimum version of cyrus sources can I use for these changes?
>> At the moment I'm running servers with 2.4.12, on our illumos
>> based distro XStreamOS.
>>
>> ----------------------------------------------------------------------------------------
>> *Sonicle S.r.l. *: http://www.sonicle.com
>> *Music: *http://www.gabrielebulfon.com
>> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
>>
>> ------------------------------------------------------------------------
>>
>>
>> *Da:* Ken Murchison <murch at andrew.cmu.edu>
>> *A:* gbulfon at sonicle.com info-cyrus at lists.andrew.cmu.edu
>> *Data:* 14 luglio 2016 17.14.18 CEST
>> *Oggetto:* Re: thread=refs
>>
>>
>> I went ahead and committed an implementation of THREAD-REFS:
>> https://github.com/cyrusimap/cyrus-imapd/commit/16747e608f32f9dd9348988d3f83cb8f1b037ff6
>>
>> Per the draft, grouping by subject is skipped and threads
>> (toplevel messages) are sorted by INTERNALDATE, while the
>> messages within the threads are still sorted by SENTDATE.
>>
>> I confirmed that THREAD=REFERENCES is still correct, but I
>> have nothing to compare THREAD=REFS results to. The current
>> threading in Thunderbird is close, but it might be using
>> INTERNALDATE throughout.
>>
>>
>> On 07/12/2016 04:44 PM, Ken Murchison via Info-cyrus wrote:
>>> Gabriele,
>>>
>>> The attached patch is what I was thinking in terms of
>>> implementation. It skips the grouping by subject for REFS,
>>> but I didn't do the REFS-specific date handling. Contrary to
>>> what the THREAD=REFS draft says, I'm not sure if the special
>>> date handling should be done in step 4 or 6. I would have to
>>> did deeper into the code to see where is belongs.
>>>
>>>
>>> On 07/08/2016 11:36 AM, Gabriele Bulfon via Info-cyrus wrote:
>>>> Mainly web clients, installed clients usually implements
>>>> threading internally to overcome problems with the original
>>>> references algorithm that is often confusing.
>>>>
>>>> The problem with references is that it includes subject
>>>> grouping, that is an old netscape model of the 90s: today,
>>>> we just need references within the headers ids, or we may
>>>> take a wrong message in the thread just because it has a
>>>> similar subject (for example automatic mails with same
>>>> subjects would be treated as a thread, which is wrong).
>>>>
>>>> Now, we're staring to implement threading view on our web
>>>> collaboration software, running on cyrus.
>>>> So we are investigating how RoundCube is doing threading on
>>>> a dovecot installation, and we found it to be the same as
>>>> the client algrithm of Thunderbird, which is fine. Looking
>>>> at the protocol, it uses REFS first, probably because it
>>>> has no subject grouping by definition, and it should have a
>>>> better date sorting. Should, because I found that Dovecot
>>>> does not sort it reversed...
>>>>
>>>> Maybe I will ask Dovecot guys why they choose to keep sort
>>>> same as references: I suspect that claim to support refs,
>>>> but actually the do the same references functions, but
>>>> never do subject grouping.
>>>>
>>>> ----------------------------------------------------------------------------------------
>>>> *Sonicle S.r.l. *: http://www.sonicle.com
>>>> *Music: *http://www.gabrielebulfon.com
>>>> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> *Da:* Ken Murchison <murch at andrew.cmu.edu>
>>>> *A:* gbulfon at sonicle.com info-cyrus at lists.andrew.cmu.edu
>>>> *Data:* 8 luglio 2016 17.17.32 CEST
>>>> *Oggetto:* Re: thread=refs
>>>>
>>>>
>>>>
>>>>
>>>> On 07/08/2016 11:08 AM, Gabriele Bulfon wrote:
>>>>> Ok, sure, but still two issues remain other than the
>>>>> draft:
>>>>> - we need to get rid of subject grouping in REFS, it
>>>>> only brings disorder, merging stuff that is not related
>>>>
>>>> I believe that the parameterization of the core
>>>> functions should be able to handle this.
>>>>
>>>>
>>>>> - I would try to guess why dovecot does not change
>>>>> sorting in REFS, keeping it same as REFERENCES
>>>>
>>>> I would contact that Dovecot authors and find out which
>>>> version of the THREAD=REFS draft they based their work on.
>>>>
>>>> BTW, which clients use THREAD=REFS?
>>>>
>>>>
>>>>>
>>>>> ----------------------------------------------------------------------------------------
>>>>> *Sonicle S.r.l. *: http://www.sonicle.com
>>>>> *Music: *http://www.gabrielebulfon.com
>>>>> *Quantum Mechanics :
>>>>> *http://www.cdbaby.com/cd/gabrielebulfon
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> *Da:* Ken Murchison <murch at andrew.cmu.edu>
>>>>> *A:* gbulfon at sonicle.com info-cyrus at lists.andrew.cmu.edu
>>>>> *Data:* 8 luglio 2016 16.39.17 CEST
>>>>> *Oggetto:* Re: thread=refs
>>>>>
>>>>>
>>>>> Is there an actual RFC? All I find is
>>>>> draft-ietf-morg-inthread-01. Looking at that
>>>>> draft, the only difference between REFS ad
>>>>> REFERENCES is this:
>>>>>
>>>>> THREAD=REFS sorts threads by the most recent INTERNALDATE in each
>>>>> thread, replacing THREAD=REFERENCES step (4). This means that when a
>>>>> new message arrives, its thread becomes the latest thread. (Note
>>>>> that while threads are sorted by arrival date, messages within a
>>>>> thread are sorted by sent date, just as for THREAD=REFERENCES.)
>>>>>
>>>>>
>>>>> This being the case, I don't think we need two
>>>>> copies of the threading functions. I'd modify the
>>>>> exiting functions to take an additional parameter
>>>>> to specify whether we're doing REFS or REFERENCES
>>>>> and then have 2 wrapper functions which call the
>>>>> main function with the parameter set appropriately
>>>>> for the given algorithm.
>>>>>
>>>>>
>>>>> On 07/08/2016 10:03 AM, Gabriele Bulfon wrote:
>>>>>> Ok, it works :) but checking against dovecot
>>>>>> implementation, it looks like they have refs
>>>>>> order same as references, but without subject
>>>>>> grouping. AFAIK the RFC on refs says ordering of
>>>>>> dates within the group should be reversed. Am I
>>>>>> wrong?
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------
>>>>>> *Sonicle S.r.l. *: http://www.sonicle.com
>>>>>> *Music: *http://www.gabrielebulfon.com
>>>>>> *Quantum Mechanics :
>>>>>> *http://www.cdbaby.com/cd/gabrielebulfon
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> *Da:* Gabriele Bulfon via Info-cyrus
>>>>>> <info-cyrus at lists.andrew.cmu.edu>
>>>>>> *A:* Ken Murchison <murch at andrew.cmu.edu>
>>>>>> info-cyrus at lists.andrew.cmu.edu
>>>>>> *Data:* 8 luglio 2016 15.22.56 CEST
>>>>>> *Oggetto:* Re: thread=refs
>>>>>>
>>>>>>
>>>>>> Testing ;) and checking against a dovecot
>>>>>> machine with refs and same messages.
>>>>>> Will let you know
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------
>>>>>> *Sonicle S.r.l. *: http://www.sonicle.com
>>>>>> *Music: *http://www.gabrielebulfon.com
>>>>>> *Quantum Mechanics :
>>>>>> *http://www.cdbaby.com/cd/gabrielebulfon
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> *Da:* Ken Murchison <murch at andrew.cmu.edu>
>>>>>> *A:* gbulfon at sonicle.com
>>>>>> info-cyrus at lists.andrew.cmu.edu
>>>>>> *Data:* 8 luglio 2016 15.02.38 CEST
>>>>>> *Oggetto:* Re: thread=refs
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 07/07/2016 02:03 PM, Gabriele Bulfon
>>>>>> via Info-cyrus wrote:
>>>>>>> I can finally get back to this after so
>>>>>>> many months!
>>>>>>> I checked the sources, and I actually
>>>>>>> see it doesn't look very hard.
>>>>>>>
>>>>>>> Looks like:
>>>>>>> - renaming all functions like
>>>>>>> "index_thread_ref" into
>>>>>>> "index_thread_references"
>>>>>>> - duplicate them as "index_thread_refs"
>>>>>>> - let "references" alg call the
>>>>>>> "references" funcs
>>>>>>> - add support for "refs" in thread_algs
>>>>>>> and let them call the "refs" funcs
>>>>>>
>>>>>> Makes sense.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> then:
>>>>>>> - completely remove the call to
>>>>>>> "ref_group_subjects", we don't want it
>>>>>>> at all in refs
>>>>>>> - change the sortcrit to use the
>>>>>>> SORT_REVERSE modifier
>>>>>>
>>>>>> As long as you mean making these changes
>>>>>> for just the "refs" variant and not both.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> what do you think? may be fine?
>>>>>>>
>>>>>>> ----------------------------------------------------------------------------------------
>>>>>>> *Sonicle S.r.l. *: http://www.sonicle.com
>>>>>>> *Music: *http://www.gabrielebulfon.com
>>>>>>> *Quantum Mechanics :
>>>>>>> *http://www.cdbaby.com/cd/gabrielebulfon
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> *Da:* Ken Murchison <murch at andrew.cmu.edu>
>>>>>>> *A:* info-cyrus at lists.andrew.cmu.edu
>>>>>>> *Data:* 5 ottobre 2015 14.04.02 CEST
>>>>>>> *Oggetto:* Re: thread=refs
>>>>>>>
>>>>>>>
>>>>>>> As far as I can tell, the last
>>>>>>> specification for thread=refs was
>>>>>>> here:https://tools.ietf.org/html/draft-ietf-morg-inthread-01
>>>>>>>
>>>>>>> To implement this you want to look
>>>>>>> at index.c in the Cyrus source and
>>>>>>> add another entry to the
>>>>>>> thread_algs[] array. I'm guessing
>>>>>>> that you can reuse a lot of the
>>>>>>> existing index_thread_ref() code
>>>>>>> (which is probably needs to be
>>>>>>> renamed to index_thread_references()).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/05/2015 06:07 AM, Gabriele
>>>>>>> Bulfon wrote:
>>>>>>>> Great, Ken. Can you give me some
>>>>>>>> advice / pointer to the sources I
>>>>>>>> should look at?
>>>>>>>>
>>>>>>>> Gabriele
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> *Da:* Ken Murchison
>>>>>>>> <murch at andrew.cmu.edu>
>>>>>>>> *A:* info-cyrus at lists.andrew.cmu.edu
>>>>>>>> *Data:* 2 ottobre 2015 19.08.04 CEST
>>>>>>>> *Oggetto:* Re: thread=refs
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/02/2015 10:53 AM,
>>>>>>>> Gabriele Bulfon wrote:
>>>>>>>>> Nice, it's not a big deal for
>>>>>>>>> us to upgrade to new versions,
>>>>>>>>> surely easier than porting to
>>>>>>>>> Dovecot! ;)
>>>>>>>>>
>>>>>>>>> So, maybe we can help with the
>>>>>>>>> implementation.
>>>>>>>>> In my mind, it's almost about
>>>>>>>>> changing the
>>>>>>>>> "thread=reference" and let it
>>>>>>>>> omit the subject matching,
>>>>>>>>> change sorting
>>>>>>>>> and...maybe just this? How
>>>>>>>>> much hard do you think it is?
>>>>>>>>>
>>>>>>>>
>>>>>>>> That sounds about right from
>>>>>>>> what I remember of
>>>>>>>> THREAD=REFERENCES (which I
>>>>>>>> co-authored and implemented)
>>>>>>>> and THREAD=REFS (which I think
>>>>>>>> was last documented in 2010).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Da:* Bron Gondwana
>>>>>>>>> <brong at fastmail.fm>
>>>>>>>>> *A:*
>>>>>>>>> info-cyrus at lists.andrew.cmu.edu
>>>>>>>>> *Data:* 2 ottobre 2015
>>>>>>>>> 12.59.08 CEST
>>>>>>>>> *Oggetto:* Re: thread=refs
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No, there isn't. The
>>>>>>>>> conversations work in 3.0
>>>>>>>>> beta contains a lot of
>>>>>>>>> what would be required to
>>>>>>>>> efficiently implement
>>>>>>>>> THREAD=REFS, but nobody
>>>>>>>>> has done the work to
>>>>>>>>> implement it.
>>>>>>>>> It certainly will never be
>>>>>>>>> backported to the 2.4
>>>>>>>>> series, which is only
>>>>>>>>> getting security updates
>>>>>>>>> and fixes for major bugs now.
>>>>>>>>> Regards,
>>>>>>>>> Bron.
>>>>>>>>> On Fri, Oct 2, 2015, at
>>>>>>>>> 18:40, Gabriele Bulfon wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> we have systems running
>>>>>>>>>> cyrus 2.4.12, where
>>>>>>>>>> thread algorithms are
>>>>>>>>>> only references and
>>>>>>>>>> orderedsubject.
>>>>>>>>>> Is there support for the
>>>>>>>>>> thread=refs algorithm?
>>>>>>>>>> Thanks
>>>>>>>>>> Gabriele
>>>>>>>>>> ----
>>>>>>>>>> Cyrus Home Page:
>>>>>>>>>> http://www.cyrusimap.org/
>>>>>>>>>> List Archives/Info:
>>>>>>>>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>>>>>> To Unsubscribe:
>>>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>>>>> --
>>>>>>>>> Bron Gondwana
>>>>>>>>> brong at fastmail.fm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----
>>>>>>>>> Cyrus Home Page:http://www.cyrusimap.org/
>>>>>>>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>>>>> To Unsubscribe:
>>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kenneth Murchison
>>>>>>>> Principal Systems Software Engineer
>>>>>>>> Carnegie Mellon University
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----
>>>>>>>> Cyrus Home Page:http://www.cyrusimap.org/
>>>>>>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>>>> To Unsubscribe:
>>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>>>
>>>>>>> --
>>>>>>> Kenneth Murchison
>>>>>>> Principal Systems Software Engineer
>>>>>>> Carnegie Mellon University
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----
>>>>>>> Cyrus Home Page:http://www.cyrusimap.org/
>>>>>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>>> To Unsubscribe:
>>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>>
>>>>>> --
>>>>>> Kenneth Murchison
>>>>>> Principal Systems Software Engineer
>>>>>> Carnegie Mellon University
>>>>>>
>>>>>> ----
>>>>>> Cyrus Home Page:http://www.cyrusimap.org/
>>>>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>>> To Unsubscribe:
>>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>>>
>>>>>
>>>>> --
>>>>> Kenneth Murchison
>>>>> Principal Systems Software Engineer
>>>>> Carnegie Mellon University
>>>>>
>>>>
>>>> --
>>>> Kenneth Murchison
>>>> Principal Systems Software Engineer
>>>> Carnegie Mellon University
>>>>
>>>>
>>>>
>>>> ----
>>>> Cyrus Home Page:http://www.cyrusimap.org/
>>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>
>>> --
>>> Kenneth Murchison
>>> Principal Systems Software Engineer
>>> Carnegie Mellon University
>>>
>>>
>>> ----
>>> Cyrus Home Page:http://www.cyrusimap.org/
>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>> --
>> Kenneth Murchison
>> Principal Systems Software Engineer
>> Carnegie Mellon University
>>
>
> --
> Kenneth Murchison
> Principal Systems Software Engineer
> Carnegie Mellon University
>
--
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20160719/26b0cb35/attachment-0001.html>
More information about the Info-cyrus
mailing list