cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Chris Stromsoe]

On Tue, 23 Sep 2003, Pat Lashley wrote:

> --On Monday, September 22, 2003 22:40:00 -0700 Chris Stromsoe
> <cbs at cts.ucla.edu> wrote:
>
> I keep mentioning Envelope-To because 30+ years of Software Engineering
> experience tells me that if you handle Return-Path, people will also
> want Envelope-To.

out of ~40,000 messages I've got handy, between 50 and 60 had some variant
of envelope-to (either plain or with x- or old-), and 36,906 had a
return-path header.  Return-Path is an rfc mandated header that must be
handled.  I'm still not sure how Envelope-To applies to this discussion.

> > That, imho, is not correct behavior.
>
> What headers, aside from Return-Path, are not cached in RAM?

The Received header that lmtpd adds and the X-Sieve header.  There may be
others in other areas of the code, but those two and the Return-Path
header are the three that I saw.

> > I don't what the Sieve RFC says about return-path.  I'm pretty sure
> > that it isn't germane to this topic.
>
> Then why was it necessary to create an "envelope" extension to sieve?

I don't know.  Maybe early versions of lmtpd weren't following rfc2821 and
properly settings the return-path, and sieve couldn't get at the
information any other way.  Maybe the envelope extension was added to get
at envelope-to and envelope-from was added as an afterthought.

> > smartsieve can't be smart enough to automatically translate the
> > condition. It seems like you're suggesting that smartsieve (and any
> > other program that generates sieve rules) should be following behind
> > lmtpd, trying to guess at what special header might map to which
> > special sieve function.
>
> Nonsense.  I'm only suggesting that it be smart enough to actually
> follow the RFCs and drafts; and to know about a few of the extensions
> like envelope.

There is no RFC or draft that says that the return-path header should not
be visible as a header to sieve scripts, whereas 2821 does say that the
final delivery must add the return-path header.  To me, that says that
return-path should be added by lmtpd and be visible to sieve.  Making
smartsieve -- and every other application out there that generates sieve
scripts -- work around lmtpd's shortcomings doesn't make any sense to me.
Fixing lmtpd so that the in-ram cache mirrors the on-disk data store does.

> > I'm not sure what you mean here.  Or maybe I am not being clear.  To
> > restate myself:  this issue has _nothing to do with sieve_.  I am not
> > trying to patch sieve and I don't want to be.  The patch changes the
> > behavior of lmtpd so that the on-disk and in-memory represention of
> > the headers is the same.  (Rather, the patch is a start; since it
> > doesn't yet include the last-hop Received header they don't completely
> > match.)
>
> You're trying to change LMTP to provide a header that WILL NOT EXIST in
> other sieve environments.  It will not be at all obvious to the naive
> user that that header was inserted at the last minute rather than
> following the message from its inception.

I still don't see what it is that you are trying to say.  You seem to be
assuming some sort of lmtp/sieve interdependance.  Please correct me if
I'm wrong.

Other sieve environments may not have lmtp at all.  sieve is just a set of
tokens and rules for filtering.  You could probably quite easily use sieve
to parse nntp articles into mailboxes, or write a parser similar to
procmail and run sieve out of sendmail as a delivery agent.

As far as being not obvious to the naive user ... when has that stopped
anything from happening?  lmtpd rejects messages with 8-bit content,
converts characters with no character-set defined to X, will bounce mail
that has NULL characters embedded in it.  The "naive users" are doing
alright so far.

> >> The envelope extension exists to provide the functionality you need.
> >
> > The envelope extension does not provide the functionality that I need.
> > Open any message that is stored in cyrus and was delivered via lmtp.
> > Look at the headers.  The first header that you will see looks
> > something like:
> >
> >   Return-Path: <cbs at cts.ucla.edu>
> >
> > I need to be able to filter on that header.
>
> You are confusing functionality with implementation.  A common mistake.

No, I'm not:  I need the functionality of being able to filter on all of
the headers in the message.  Return-Path is a header.  lmtpd is the "final
delivery"  outside of the smtp environment.  lmtpd adds the header on
disk.  It does not cache the header in the header cache.

If there were a separate sieve parser (a la procmail) and lmtpd forked off
sieve to take action, it would act on the on-disk file and would see the
Return-Path header.  The built-in sieve parser should see the same set of
headers.


-Chris

> -Pat
>