Sieve and encoded Headers

[Bron Gondwana]

On Mon, Feb 08, 2010 at 01:03:30PM +0100, Garry wrote:
> On 08.02.2010 08:28, Bron Gondwana wrote:
> > I've CC'd Ken on this - I wonder if it's worth going back and doing a
> > "minimal still compatible" set of patches that fixes charset encoding in
> > sieve without actually changing the on disk format of the cyrus.cache
> > (which is what requires reconstructs to make things work again)
> 
> I've not looked into the sources, so please be kind if I'm totally off
> the path here, but in general I wouldn't expect the comparison to
> require changing the on-disk format of the stored information - just
> ensure that upon reading (and thus comparing) something, add a function
> that creates a decoded copy of the header string (e.g. by going to UTF8
> on everything, including the sieve rule information), then compare that ...

The cyrus.cache file stores some fields in "search format" - which
previously stripped ALL whitespace.  We changed it as part of the charset
conversion to compress all whitespace to a single SPACE ' ' character
instead so that searches wouldn't runallthewordstogetherlikethis.

That's the bit of our changes that causes a reconstruct to be necessary,
otherwise you can't search for "Bron Gondwana" any more, you need to search
for "brongondwana" instead.

The changes to charset handling are converting from a SINGLE table mapping
from any arbirary Character Set tosearchform - instead having a group of
transforms that can be piped together - so the charset mapping tables
convert to Unicode codepoints (32 bit), and then there are further stages
that can convert into search form, or into utf8, or whatever.

This allows us to re-implement the old search form easily enough (it's 
just a flag actually) if we want to.
 
> > But - I can tell you that we're not going to be backporting this sort of
> > thing to the 2.2 series, so you'll certainly need to upgrade!
> 
> I was just stating the facts, no problem here if it needs to be updated
> ... I just dropped in the current APT packages for ease of
> administration ...

Yeah, we build our own Cyrus Debian packages instead - they're not following
the FHS that well, but since Cyrus is our main thing, we don't mind having
one tool that sort of "stands alone" - also we can build up "cyrus-beta"
packages that have a different set of disk paths and just pass the correct
flags to our templates so all the config files and init scripts get built
with the correct paths built in!

Bron.