git master branch going unstable

[Bron Gondwana]

(I hope you don't mind me CC'ing this back to the list so everyone
can read it!)

On Tue, Jan 11, 2011 at 04:58:06PM +0000, Alexey Melnikov wrote:
> Hi Bron,
> 
> Bron Gondwana wrote:
> 
> >Actually - it's been unstable for a few days now since I added the
> >"specialuse" patches
> >
> I meant to ask earlier: is there a public list of things that you
> are planning in 2.5? In particular I am interested in knowing which
> IMAP extensions you and your co-workers are planning to add.

We should really discuss them on cyrus-devel and the IRC channel.
We've been quite lax about that :(

Do you have any particular extentions you're interested in?  
 
> >* conversations (cross folder thread) initial support from Greg
>
> Do you have a pointer to how this works/what does this do?

No - we'll have to write something up on this.  It's going to be
a non-standard extention to start off with.  Hopefully once we've
ironed out the details we can write an RFC.

Basically it (Greg, correct me if I've messed anything up here...):

*) adds a 64 bit "cid" (conversation id) to each index record.
*) adds a separate database (per user) which contains a map of
   message-id header to cid.
*) for each appended message, gets every "message-id" from the
   message header (Message-Id, In-Reply-To, References) and looks
   them up in the conversations database.
*) if a related message is found, it keeps the same cid.  If
   no related message is found, it allocates a "random" cid
   (first 64 bits of the sha1 of the current message in fact)
*) SPECIAL CASE: if TWO SEPARATE cids are found then we have
   a message which links two previously separate conversations.
   In this case, bias upwards - choose the higher cid and re-number
   all the messages with the lower cid.
   (the same renumbering logic will be used by replication to merge
    different cids from different ends)

There's a bit more magic to do with remembering which folders
are "touched" by each conversation to make the implementation more
efficient.  And of course making reconstruct work, and able to
repair a damaged conversations database.

.....

We're about here now.  The next step is defining a good interface to
access this data :)  

In particular, there's a few things we need:

>From Folder/UidValidity/Uid triplet - get a conversation ID.

>From a triplet or a conversation ID - get a list of all "related"
messages (those with the same conversation ID) across ALL folders
belonging to the same user.

SEARCH - within a conversation
SORT - within a conversation

.....

Kind of somewhat separately, we want some form of sorted search
windowing... where you can say "give me the first 10 messages which
match this search within this sort - and it will sort the messages
(efficient) first and then apply the search (usually more expensive)
to each message until it's found enough matches - then return!

At the moment it always searches first, then sorts the search results,
which can be very expensive for a body search across all folders - 
which is what customers often want, funnily enough.

Bron.