How to make cyrus not change non US-ASCII characters to "X"

[Christian Schulte]

Alessandro Oliveira wrote:

> Maybe the best solution would be a filter to change the message body 
> encoding using quoted-printable instead of putting X everywhere, and 
> change headers to also use the appropriate encoding. What do you think 
> about this ?
>
> Christian Schulte escreveu:
>
>>> Is there any possibility to put this as an autoconf option?
>>>
>>> maybe something like: "./configure --enable-strict-headers | 
>>> --disable-strict-headers" would do the trick,
>>> or even better, a runtime configuration option !!!
>>>
>>> what are the REAL consequences of having 8bit characters in the 
>>> Subject header ?
>>>
>>>>> A> Date: Wed, 20 Nov 2002 09:30:31 -0200
>>>>> A> To: info-cyrus <info-cyrus at lists.andrew.cmu.edu>
>>>>>
>>>>> A> Hi,
>>>>>
>>>>> A> I'd like to know if there is a way to avoid this cyrus 
>>>>> behavior, my
>>>>> A> users are getting very angry with it.
>>>>>
>>>>> A> Thanks for any help,
>>>>>
>>>>> Yes, users do. And notice about strict 7-bit ASCII within headers
>>>>> does not help.
>>>>>
>>>>> Same problem using Cyrillic. I have not found anything else and
>>>>> used a simple hack. Locate the following 2 files in the source 
>>>>> tree: lmtpengine.c and message.c
>>>>>
>>>>> change them the following way ( by inserting `#if 0' and `#endif' 
>>>>> ) and re-compile the entire software.
>>>>>
>>>>> This eliminates the substitution unconditionally.
>>>>>
>>>>>
>>>>> lmtpengine.c:
>>>>>
>>>>> .....
>>>>>                /* ignore this whitespace, but we'll copy all the 
>>>>> rest in */
>>>>>                break;
>>>>>            } else {
>>>>>
>>>>> // HACK -- Violet
>>>>> #if 0
>>>>>                if (c >= 0x80) {
>>>>>                    if (reject8bit) {
>>>>>                        /* We have been configured to reject all 
>>>>> mail of this
>>>>>                           form. */
>>>>>                        r = IMAP_MESSAGE_CONTAINS8BIT;
>>>>>                        goto ph_error;
>>>>>                    } else {
>>>>>                        /* We have been configured to munge all 
>>>>> mail of this
>>>>>                           form. */
>>>>>                        c = 'X';
>>>>>                    }
>>>>>                }
>>>>> #endif
>>>>>                /* just an ordinary character */
>>>>>                body[off++] = c;
>>>>> .....
>>>>>
>>>>> and message.c:
>>>>>
>>>>> .....
>>>>>            else {
>>>>>                sawcr = 0;
>>>>>                blankline = 0;
>>>>> // HACK -- Violet
>>>>> #if 0
>>>>>                if (inheader && *p >= 0x80) {
>>>>>                    if (reject8bit) {
>>>>>                        /* We have been configured to reject all 
>>>>> mail of this
>>>>>                           form. */
>>>>>                        if (!r) r = IMAP_MESSAGE_CONTAINS8BIT;
>>>>>                    } else {
>>>>>                        /* We have been configured to munge all 
>>>>> mail of this
>>>>>                           form. */
>>>>>                        *p = 'X';
>>>>>                    }
>>>>>                }
>>>>> #endif
>>>>>            }
>>>>>        }
>>>>>
>>>>>        fwrite(buf, 1, n, to);
>>>>>    }
>>>>> .....
>>>>>
>>>> In 2.2 cvs you do this in message.c and spool.c
>>>
>>>
>> I thought that cyrus does not only replace non US-ASCII characters in 
>> the mail-header. If a mail with a specified enconding of US-ASCII 
>> gets send through LMTP all non US-ASCII characters in the mail body 
>> get replaced to "X" also! My users were not interested in this 
>> behaviour if that would only affect the header. The problem is that 
>> in e.g. germany all umlauts like ä,ü,ö etc. get replaced in the body 
>> if specified as US-ASCII. Of course the mail body specifies a wrong 
>> character encoding if sent as US-ASCII but containing umlauts but 
>> even a big free-mail-provider (www.web.de) in germany with millions 
>> of users sends his newsletter in US-ASCII but with umlauts! I got it 
>> done with these small source-changes but this will not change user 
>> from sending mail in wrong encodings :-(
>
>
That are things which can be done with sendmail before delivering to 
LMTP but I did not get it done right now! The convert-code in sendmail 
is quite buggy in 8.12.6 and cannot reliably convert encodings as well. 
Adding such features to lmtpd in cyrus would be great but I think that 
this is not a asap-topic.