APPEND vs RFC2822 vs STD0011

[Edward Reid]

The mail provider (MX) for my domain, fastmail.fm, runs cyrus. I use
Eudora (for Mac, v5.2), mostly in POP mode, but I use some IMAP
features too. In particular, some of my filters copy incoming (POP)
messages to an IMAP mailbox at fastmail.fm. That's where the problems
start.

Some of these incoming messages contain NULs or bare CR or LF. Yes, the
sender is broken as far as RFC2822 is concerned, but the messages get
through anyway. And the messages are valid RFC822/STD0011 format.

When Eudora tries to copy these (APPEND them) to the IMAP mailbox,
cyrus (at fastmail.fm) returns an error. I could live with an
occasional copy failure, but the worst part is that when Eudora gets
the server error, it thinks it's a terrible problem and throws up a
dialog box and ceases all processing. Since I (like many people) depend
on Eudora cleaning up my mailbox and doing other things with incoming
mail automatically when I'm not at my desk, this gets to be a serious
problem.

So I started reading RFC3501 to find the reason. I assumed that I'd
find a good reason that I could quote to Eudora support, telling them
why Eudora has to clean up the message before storing it in an IMAP
mailbox. But I didn't find that. What I found -- under the APPEND
command (section 6.3.11) -- is

: The APPEND command appends the literal argument as a new message
: to the end of the specified destination mailbox. This argument SHOULD
: be in the format of an [RFC-2822] message.

Note well: that's "SHOULD", not "MUST". This is important. RFC2119
gives the meaning of SHOULD:

: This word [...] mean[s] that there may exist valid reasons in
: particular circumstances to ignore a particular item [...]

So based on my reading of the RFC, it's the client's choice: it should
normally append RFC2822 messages, but if it has a valid reason, it's
allowed to append something that's not RFC2822. Now, IMAP mailboxes are
intended for email -- "Internet message format" or "Internet text
messages" in the RFC language -- and so it would be hard to make a case
for storing anything that's not such a message. But RFC822 messages are
still rampant on the Internet. In fact, as I understand it, although
RFC2822 has obsoleted RFC822, STD0011 (which is identical to RFC822) is
still a standard and has not yet been superseded.

And it certainly seems to me that making a copy of an existing message
is a "valid reason" for copying it intact, without the modifications
needed to force it to conform to the stricter format of RFC2822. Since
RFC3501 leaves this decision up to the client, it follows that cyrus is
broken when it refuses the message.  If RFC3501 said "MUST", then I'd
say it's Eudora's responsibility to fix the message before attempting
an APPEND. But the RFC says "SHOULD".

Is there any good argument for cyrus' action? If there is, I'd be happy
to take it to Eudora and push them to fix Eudora. Eudora's not exactly
known for its stellar IMAP support, and I'd like them to improve this.
I've shoved the RFCs in their face plenty of times in the past. But in
this case, my reading of the RFCs is that Eudora's APPEND action is
defensible and cyrus' action is incorrect.

I'm very interested in hearing the cases for both sides.

Edward