Fwd: Huge header detection

[Carlos Horowicz]

On Mon, Feb 9, 2009 at 8:53 PM, Carlos Horowicz
<carlos.horowicz at gmail.com> wrote:
> On Mon, Feb 9, 2009 at 9:02 AM, Bron Gondwana <brong at fastmail.fm> wrote:
>> On Mon, Feb 09, 2009 at 10:24:38AM +0000, Ian Eiloart wrote:
>>>
>>>
>>> --On 7 February 2009 02:36:36 -0200 Carlos Horowicz
>>> <carlos.horowicz at gmail.com> wrote:
>>>
>>>> I'm wondering what to do in a live system with may be hundreds of
>>>> thousands of these strange e-mails already in users´ mailboxes,
>>
>> My god, that many?  Nuke it from orbit.  Only way to be sure.
>>
>> (alternative plan, grep for the bogus files, unlink them and
>> reconstruct the mailboxes - cheaper and less radioactive)
>
> I have already done that (find ... -exec grep ...) but it took days to
> scan entire volumes, and in this case I already knew which pattern to
> look for.
>
>>
>>>> Should imapd be patched so that it just ignores the repetitions , both
>>>> when building cyrus.cache and when it returns the headers to a client
>>>> ? or should imapd really modify the original e-mail by stripping
>>>> unnecessary/illegal headers and store a cleaned-up version ?
>>>
>>> It shouldn't be modifying messages. It should handle such messages
>>> elegantly. Ignoring repetitions (beyond a threshold of repeats) seems the
>>> most sensible option. However, failing to report them to a client could
>>> cause confusion, so a threshold should be reasonably high. Of course some
>>> headers are supposed to have multiple instances...
>>
>> Ditto with that.  This patch ignores repetitions beyond a threshold.  It
>> turns out that ignoring specific headers separately is hard[tm], but
>> just stopping parsing them after a count is way-easy.
>>
>>> Alerting the system administrator to the existence of such bogus messages
>>> seems like a good idea, too. Perhaps through the logging system.
>>
>> Yeah, good point.  Allow me to add that.
>>
>>> If you don't want a particular message in the system, then it should not
>>> be accepted by LMTP or by any IMAP message creation mechanism.
>>
>> Ditto.  Gosh.  That makes 3 tunables.  The gods of tunable
>> non-proliferation will want my head for doing this:
>>
>> maxcacheheaders_warn = 500
>> maxcacheheaders_skip = 1000 (same as the current patch)
>> maxcacheheaders_reject = 2000
>>
>> Sound like reasonable defaults?  I'm tempted to make the _reject be '0'
>> (don't reject) by default.
>
>
> Is this patch intended to be also part of reconstruct ? 'cause this
> would catch many other situations, like restores, spool rebuilds, and
> even scanning imap spools.

I was thinking of reconstruct just for the sake of warning the admin.

for instanace in
syslog(LOG_ERR, "ERROR: message has %d header lines, not caching any
more", max_skip);

how can the message be identified so the admin can cat upon ?





>
>>
>> Bron.
>>
>
> Carlos
>