How to get every message (even a malformed one) onto Cyrus imapd?

[Recursive]

Niels,

thank you very much for taking the time.

On 28.05.2015 11:10, Niels Dettenbach wrote:
> if a "kind of a message" is hardly violating any protocol standards, from some 
> point of level it ISN't a mail anymore. I did not recognized in the decade(s) 
> of using cyrus lmtp that cyrus did not accepted any "real" email. So it would 
> be nice to get any details about what type of records / data is "blocked" by 
> cyrus LMTP in your situation.

You are right. Unfortunately, I have put a lot of time into trying to debug the situation, without any success so far (see my second post to this list from today). Moreover, the messages in question were originally received by providers like GMX, meaning that I can read them using a webmail UI, but that I can't view the full headers (unless I have missed some button or menu entry).

> "Spam" is not "malformed" usually today - so this is another story. But an 
> email is an email, if it fit's ALL points of the protocol requirements (by 
> RFC) - if not you or any other mail system is not able to handle a message 
> really properly and correctly. For "legal reasons" you can use proper logging 
> here, proving what was done (and what not) on your mail systems side.

You are completely right again regarding the definition of what an email is. But I am not sure if non-technical persons or organizations, notably judges and lawyers, see things like we do. Thus, I'd really like to avoid arguing that way.

Regarding logging, it would make things too complicated for me. I would have to keep the logs as long as the relevant messages themselves, i.e. 10 years at least.

All other solutions that came to my mind are very uncomfortable (and thus unreliable) as well (for example, I could open the messages via webmail UI, copy their content and store it into files in my local file system). I really can't afford to do such things manually, and I can't afford to look into every of my email accounts via webmail UI on a daily basis.

Rather, I am looking for a general solution to this problem which works automatically.

> The very most of (professional) internet-connected email systems are not 
> accepting mails which are not fulfilling the basic standards even if they are 
> "able" to "reach" the MX from anywhere and/or have a proper Rcpt To - by very 
> good reasons (i.e. at least to avoid abusive usage of your MX by third 
> parties).

I could really imagine that the messages in question have been sent by unusual spamming software of by spammers who are too silly to operate their software in the right way. But there is no doubt that I (at the very moment) have 80 messages in a GMX account (all about 16 months old) which all cause the problem I have described.

This means that these messages well-formed enough for GMX to accept them, but are so malformed that cyrus lmtp / imapd rejects them.

> If you are willing to accept even stuff which is not fulfilling the basic 
> requirements you have to make shure that the stuff is converted to any real 
> mail so that serviecs or sofwtare afterwards is able to handle it as real 
> email. If it is true that any kind of headers are not accepted - just let 
> rewrite (and for "legal reasons" may be log that) it.

Yes, I'd really like to accept any stuff.

For me, in such cases, it would be perfectly acceptable to let the MDA convert the whole (defective) message into an attachment, let it generate a new (well-formed) message (possibly based on some header template), let it add the attachment to that message, and let it deliver the whole thing to the respective account. Is it possible to do such things using cyrus lmtpd / imapd?

Rewriting would be perfectly acceptable as well. But unfortunately, I don't have any clue which message part I should let rewrite (because I don't know what causes the problem), and furthermore, since I am looking a general solution, I would need a method which could dynamically deal with every header (more general, with every message part) which cyrus lmtpd / imapd could reject. Additionally, rewriting would have only to take place in case of defective messages.

I don't have any experience with server side filtering / header rewriting. So I would be grateful if you could tell me if there is an easy way to rewrite the headers if the defective header(s) is (are) not known in advance, and if you could provide a pointer to respective documentation.

Thank you very much again,

Recursive