Cyrus Patches used at FastMail.FM

[Bron Gondwana]

On Sat, 27 May 2006 02:06:20 -0300, "Henrique de Moraes Holschuh" <hmh at debian.org> said:
> On Sat, 27 May 2006, Robert Mueller wrote:
> > I think having this as an *option* is perfectly reasonable. There are 
> 
> So do I, Debian has this patch for three or four years now. It comes with
> a
> big warning to the likes of "You are stupid to enable this and still
> wonder
> why multi-language search is now broken in your server".
> 
> But this has never been the instance on this issue for upstream inclusion
> of
> this patch, and I was (and still am) quite surprised it was included
> without
> any further comment.

Ph33r the power of my 'l33t bamboozling skills.

> > 3. Accept the message and store as is, leave it up to the email client to 
> > handle - This seems to be the 90% good enough that works well. Most clients 
> 
> This breaks IMAP SEARCH, or so I have been told for four years, now.
> 
> This is why I want a better explanation now for the inclusion of such a
> patch  :-)

Well, it may be worth adding more comments to the config option making
it clear that it breaks IMAP SEARCH.  I'd certainly support that.

> > 4. Recode correctly in MIME encoding based on the message charset - Would 
> > be great but:
> 
> Nah, just according to some site-wide config option from imap.conf would
> be
> more than good enough.

Wow, I'm glad you know enough about everyone else's situtation to
be able to make a blanket determination like that.  We have users
from all over the world, using:

  COUNT(DISTINCT DefCharSet)
  59

59 different charactersets - all on the same set of servers.  I
don't know any characterset that would be sufficient to put in
imapd.conf that would give the expected results for all of them.
It would be lovely if that was the case, or everyone was using
UTF-8 (except I'd likely be out of a job because things would
actually "just work[tm]")

> > I agree that this option is truly a hack, and technically the email is 
> > non-RFC compliant so you can do whatever you want, but unfortunately out in 
> 
> This is not the issue.  The issue is whether Cyrus is corrupting its
> innards
> or not by letting this thing into the spool.

If everyone is applying the patch because the real world isn't such a
pretty place as all that, then maybe it really does belong in upstream,
especially if you get a choice at config time how much corruption you
want.  There's a reason I didn't run Courier's MTA back when I was running
Courier's IMAP server at a previous job and it's because I lost too much
mail that actual human beings were trying to send to other actual human
beings.

I'm a strong believer in both "liberal in what you accept" (unless you're
in a position to enforce a contract over an interface, in which case you
should be careful to enforce every clause) and ... how best to phrase this,
it's a bit:
 * don't alter things you don't have to.
 * don't break things you don't need to, even if they're wrong.
 * don't throw away information - even if you don't understand it.

In this case, I'm presuming the "breaks multi-language search" is an
indexing issue - and an alternative would be to skip/replace/guess the
character at indexing stage but leave the full message on disk untouched.
Tough luck if you can't search it as expected - at least you haven't
LOST information.

That last point is particularly important.  By rejecting the message out of
hand, you are preserving your pristine innards but lose interoperability
with reality.  By silently replacing unknown 8bit data with an 'X' you are
throwing away information and lying to whoever delivered it that you've
faithfully saved/reproduced what you were told.  The middle ground is to
accept the message, store it "as is" and ignore the stuff you don't
understand when building indexes.

Erm.. but I rant.  Sorry about that.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm