Fwd: Huge header detection

[Carlos Horowicz]

On Mon, Feb 9, 2009 at 9:02 AM, Bron Gondwana <brong at fastmail.fm> wrote:
> On Mon, Feb 09, 2009 at 10:24:38AM +0000, Ian Eiloart wrote:
>>
>>
>> --On 7 February 2009 02:36:36 -0200 Carlos Horowicz
>> <carlos.horowicz at gmail.com> wrote:
>>
>>> I'm wondering what to do in a live system with may be hundreds of
>>> thousands of these strange e-mails already in users´ mailboxes,
>
> My god, that many?  Nuke it from orbit.  Only way to be sure.
>
> (alternative plan, grep for the bogus files, unlink them and
> reconstruct the mailboxes - cheaper and less radioactive)

I have already done that (find ... -exec grep ...) but it took days to
scan entire volumes, and in this case I already knew which pattern to
look for.

>
>>> Should imapd be patched so that it just ignores the repetitions , both
>>> when building cyrus.cache and when it returns the headers to a client
>>> ? or should imapd really modify the original e-mail by stripping
>>> unnecessary/illegal headers and store a cleaned-up version ?
>>
>> It shouldn't be modifying messages. It should handle such messages
>> elegantly. Ignoring repetitions (beyond a threshold of repeats) seems the
>> most sensible option. However, failing to report them to a client could
>> cause confusion, so a threshold should be reasonably high. Of course some
>> headers are supposed to have multiple instances...
>
> Ditto with that.  This patch ignores repetitions beyond a threshold.  It
> turns out that ignoring specific headers separately is hard[tm], but
> just stopping parsing them after a count is way-easy.
>
>> Alerting the system administrator to the existence of such bogus messages
>> seems like a good idea, too. Perhaps through the logging system.
>
> Yeah, good point.  Allow me to add that.
>
>> If you don't want a particular message in the system, then it should not
>> be accepted by LMTP or by any IMAP message creation mechanism.
>
> Ditto.  Gosh.  That makes 3 tunables.  The gods of tunable
> non-proliferation will want my head for doing this:
>
> maxcacheheaders_warn = 500
> maxcacheheaders_skip = 1000 (same as the current patch)
> maxcacheheaders_reject = 2000
>
> Sound like reasonable defaults?  I'm tempted to make the _reject be '0'
> (don't reject) by default.


Is this patch intended to be also part of reconstruct ? 'cause this
would catch many other situations, like restores, spool rebuilds, and
even scanning imap spools.

>
> Bron.
>

Carlos