How to get every message (even a malformed one) onto Cyrus imapd?

[Recursive]

Dear all,

I am using fetchmail to get messages from various accounts into Cyrus imapd. Generally, this has been working for years now.

Recently, I took a look into my system's log files and wondered why the mail log files were growing very fast. Further analysis showed that there are email messages in some of my external server accounts which are malformed. fetchmail downloads these and tries to deliver them via lmtp to Cyrus imapd; this fails because they are malformed (probably some part of the header). fetchmail is aware of the dlivery failure and does not flush the messages on the external server (which is the correct and desired behaviour).

Consequently, the whole thing restarts with the next poll.

Now, I would like to know if there is any way to get _every_ message into cyrus via fetchmail, even if it is malformed.

There are several reasons why I want to do this, the most important ones being:

- The log files grow very fast because the poll interval is short, and I have several dozens of these messages (mainly spam) in each of my external accounts.

- For legal reasons (I am handling my professional email using this system), I must not lose any message which is sent to me, even if it is spam or it is malformed.

The latter one is the reason why I can't just delete the respective messages on the external servers.

I have put a lot of time into debugging and trying to find out what is happening there and how to get around it, but the respective error messages are worrying, and debugging cyrus does not work like it is described in the respective documents (I'll thoroughly describe this in one of my next messages because it's a different subject from what I am asking here).

My preferred solution would be one which does not involve sendmail or postfix or the like, but if using one of the "big" MTAs / MDAs is the only way to achieve what I'd like to, I'll do that.

This is on debian wheezy, cyrus imapd 2.4.16, fetchmail 6.3.21 (both probably including some debian patches which could cause changes in the behaviour that I am not aware of).

Some lines from /etc/fetchmailrc:

poll
pop.gmx.net
proto pop3
bad-header accept
user "foo at gmx.de"
ssl
pass "secret"
is "gmx_foo" here
smtphost "/var/run/cyrus/socket/lmtp"

Some lines from /etc/cyrus.conf:

START {
...
        # At least one form of LMTP is required for delivery
        # (you must keep the Unix socket name in sync with imap.conf)
        #lmtp           cmd="lmtpd -a" listen="localhost:lmtp" prefork=0 maxchild=20
        lmtpunix        cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp" prefork=0 maxchild=20
...
}

The respective log entries are like that (for example, in /etc/mail.log):

May 28 09:38:04 spock fetchmail[2434]: reading message foo at gmx.de@pop.gmx.net:43 of 80 (2284 octets) (log message incomplete)
May 28 09:38:04 spock fetchmail[2434]: Connection errors for this poll:#012name 0: connection to localhost:smtp [127.0.0.1/25] failed: Connection refused.#012name 1: connection to localhost:smtp [127.0.0.1/25] failed: Connection refused.
May 28 09:38:04 spock fetchmail[2434]:  not flushed

(Note: It has turned out that this error message is quite worrying; there are definitely no connection errors with the socket. The socket exists and happily accepts all messages which are not malformed. But the worrying log message is probably fetchmail's fault, so a discussion about that would be off-topic.)

What I already have tried:

- All combinations of configuration directives "reject8bit" and "munge8bit" in /etc/imapd.conf

- Using the lmtpd daemon listening on localhost:lmtp (instead of socket) by changing /etc/cyrus.conf and changing /etc/fetchmailrc accordingly

This did not change the situation, the log messages are basically the same, and the malformed messages still don't make their way into cyrus imapd.

Starting fetchmail with the additional option --verbose leads to log entries like that:

May 27 20:53:00 spock fetchmail[18771]: reading message foo at gmx.de@pop.gmx.net:39 of 80 (2298 octets) (lo$
May 27 20:53:00 spock fetchmail[18771]: LMTP> MAIL FROM:<ucqe at harttweb.hartford.edu> SIZE=2298
May 27 20:53:00 spock fetchmail[18771]: LMTP< 250 2.1.0 ok
May 27 20:53:00 spock fetchmail[18771]: LMTP> RCPT TO:<gmx_foo at localhost>
May 27 20:53:00 spock fetchmail[18771]: LMTP< 250 2.1.5 ok
May 27 20:53:00 spock fetchmail[18771]: LMTP> DATA
May 27 20:53:00 spock fetchmail[18771]: LMTP< 354 go ahead
May 27 20:53:00 spock fetchmail[18771]: LMTP>. (EOM)
May 27 20:53:00 spock fetchmail[18771]: LMTP< 554 5.6.0 Message contains invalid header
May 27 20:53:00 spock fetchmail[18771]: Trying to connect to 127.0.0.1/25...connection failed.
May 27 20:53:00 spock fetchmail[18771]: connection to localhost:smtp [127.0.0.1/25] failed: Connection refused.
May 27 20:53:00 spock fetchmail[18771]: Trying to connect to 127.0.0.1/25...connection failed.
May 27 20:53:00 spock fetchmail[18771]: connection to localhost:smtp [127.0.0.1/25] failed: Connection refused.
May 27 20:53:00 spock fetchmail[18771]: Connection errors for this poll:#012name 0: connection to localhost:smtp [127.0$
May 27 20:53:00 spock fetchmail[18771]:  not flushed

So the reason for failure is that Cyrus lmtpd refuses the message with error code "554 5.6.0 Message contains invalid header". Unfortunately, I have found no way to find out more about this (see my next message to this list).

Thank you very much for any ideas,

Recursive