APPEND vs RFC2822 vs STD0011

[John Alton Tamplin]

Edward Reid wrote:

>The mail provider (MX) for my domain, fastmail.fm, runs cyrus. I use
>Eudora (for Mac, v5.2), mostly in POP mode, but I use some IMAP
>features too. In particular, some of my filters copy incoming (POP)
>messages to an IMAP mailbox at fastmail.fm. That's where the problems
>start.
>
>Some of these incoming messages contain NULs or bare CR or LF. Yes, the
>sender is broken as far as RFC2822 is concerned, but the messages get
>through anyway. And the messages are valid RFC822/STD0011 format.
>
>When Eudora tries to copy these (APPEND them) to the IMAP mailbox,
>cyrus (at fastmail.fm) returns an error. I could live with an
>occasional copy failure, but the worst part is that when Eudora gets
>the server error, it thinks it's a terrible problem and throws up a
>dialog box and ceases all processing. Since I (like many people) depend
>on Eudora cleaning up my mailbox and doing other things with incoming
>mail automatically when I'm not at my desk, this gets to be a serious
>problem.
>  
>
That's what sieve is for -- do it in the server and you won't have to 
rely on a particular client doing it for you.

>So I started reading RFC3501 to find the reason. I assumed that I'd
>find a good reason that I could quote to Eudora support, telling them
>why Eudora has to clean up the message before storing it in an IMAP
>mailbox. But I didn't find that. What I found -- under the APPEND
>command (section 6.3.11) -- is
>
>: The APPEND command appends the literal argument as a new message
>: to the end of the specified destination mailbox. This argument SHOULD
>: be in the format of an [RFC-2822] message.
>
>Note well: that's "SHOULD", not "MUST". This is important. RFC2119
>gives the meaning of SHOULD:
>
>: This word [...] mean[s] that there may exist valid reasons in
>: particular circumstances to ignore a particular item [...]
>
>So based on my reading of the RFC, it's the client's choice: it should
>normally append RFC2822 messages, but if it has a valid reason, it's
>allowed to append something that's not RFC2822. Now, IMAP mailboxes are
>intended for email -- "Internet message format" or "Internet text
>messages" in the RFC language -- and so it would be hard to make a case
>for storing anything that's not such a message. But RFC822 messages are
>still rampant on the Internet. In fact, as I understand it, although
>RFC2822 has obsoleted RFC822, STD0011 (which is identical to RFC822) is
>still a standard and has not yet been superseded.
>
>And it certainly seems to me that making a copy of an existing message
>is a "valid reason" for copying it intact, without the modifications
>needed to force it to conform to the stricter format of RFC2822. Since
>RFC3501 leaves this decision up to the client, it follows that cyrus is
>broken when it refuses the message.  If RFC3501 said "MUST", then I'd
>say it's Eudora's responsibility to fix the message before attempting
>an APPEND. But the RFC says "SHOULD".
>
>Is there any good argument for cyrus' action? If there is, I'd be happy
>to take it to Eudora and push them to fix Eudora. Eudora's not exactly
>known for its stellar IMAP support, and I'd like them to improve this.
>I've shoved the RFCs in their face plenty of times in the past. But in
>this case, my reading of the RFCs is that Eudora's APPEND action is
>defensible and cyrus' action is incorrect.
>  
>
I disagree - it sounds like it would be defensible if Cyrus supported 
storing such messages even though it is clearly recommended against by 
the standard.  If Eudora insists on storing such messages, it should be 
prepared to deal with a server that is unwilling to do so.

If you allow the IMAP server to store arbitrary data, it makes the other 
functions much more difficult -- if it can't assume there is a 
message-id that is globally unique, it has to create its own unique key 
for a message.  Searching based on header fields is more problematic 
since you can't assume there is even a header or that if there is one 
that it's value corresponds to anything you might expect (ie, character 
set, line length, and other issues).

Allowing null characters in particular is problematic for any code that 
uses null-terminated strings for messages or parts of messages, and 
would require changing the code everywhere to use and pass the length of 
all the strings instead.  I don't know if there is a technical reason 
behind not supporting bare newlines or not.

As far as STD0011 not being obsoleted, there are plenty of RFCs etc. 
that are not obsoleted by something but are still not best current 
practice.  Clearly if the RFC it has been based on has been obsoleted, 
the STD should be updated as well.

-- 
John A. Tamplin                               Unix System Administrator
Emory University, School of Public Health     +1 404/727-9931