RFC: Charset Conversion Routines

[Alexey Melnikov]

Bron Gondwana wrote:

>I'm in the process of rewriting the lib/mkchartable.c 
>and lib/charset.c with the eventual goal being a more
>flexible charset conversion API that can be used to
>make sieve rules match on the decoded values, and
>other funky things.
>
>It turns out to be quite a lot of changes.  My initial
>work in progress is up here:
>
>http://github.com/brong/cyrus-imapd/commit/863b5b51dd27f184fa00de4ec5a6aca3308fc30e
>
>As you can see, it's quite a bit of code.
>
>
>Anyway - I'd like some feedback on a couple of things:
>
>a) It's going to use a little more CPU this way, because
>   instead of having a table that converts _directly_ from
>   the source charset to utf-8 in search-canonical-form, 
>   it does one conversion to unicode characters (16bit), 
>   then another table converts that into a stream of zero
>   to 15 characters (yes, something expands to 15 separate 
>   codepoints, no, I don't want to know what it is!)  
>
>   Finally a third pass converts to utf-8 from the
>   character codepoints.
>
>b) Should we make this 32bit unicode characters while we're
>   at it, and extend the UTF-8 converter?
>  
>
Yes!
And upgrade the tables to Unicode 5.1.0.
And also change the normalization to conform to RFC 5051.

>c) For that matter, should we just be outsourcing all this
>   crap to another library?  Does anyone know a good library
>   that can do what Cyrus does (take one character at a time
>   and keep state?)
>  
>
I am not sure about that, but if people know a good library...