cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Chris Stromsoe]

On Tue, 23 Sep 2003, Pat Lashley wrote:

> --On Tuesday, September 23, 2003 02:18:50 -0700 Chris Stromsoe
> <cbs at cts.ucla.edu> wrote:
>
> > out of ~40,000 messages I've got handy, between 50 and 60 had some
> > variant of envelope-to (either plain or with x- or old-), and 36,906
> > had a return-path header.  Return-Path is an rfc mandated header that
> > must be handled.
>
> None of the messages should have had either of those headers when you
> recieved them.

It's very unlikely that any of the messages did have a Return-Path when
they were handed to me.  I wasn't saying that they did.  All of the
messages containing envelope-to variants came that way.  (And, in fact,
there is no reason why they shouldn't -- as you said yourself, envelop-to
is not a standard header; it could be inserted at either end.)

> Yes, the RFC mandates a Return-Path header; but it specificly says
> that it is to be generated only at the point of final delivery into
> a file.

Keep reading the RFC.  It also says that the message is considered
delivered when it leaves the SMTP environment.  Ie, when lmtpd takes it,
lmtpd should add the Return-Path header.

> >> I keep mentioning Envelope-To because 30+ years of Software
> >> Engineering experience tells me that if you handle Return-Path,
> >> people will also want Envelope-To.

....

> >        I'm still not sure how Envelope-To applies to this discussion.
>
> Please re-read the first quoted paragraph above.  Repeat until you
> DO understand how Envelope-To applies to this discussion...

This seems very much like you're sniping at me.  I almost expected the
sentence to end with ", junior" or something similar.  It's nice that you
have a lot of experience doing software development and that you bring
your experience to play.  However, I don't believe that your experience
has anything to do with this issue.

The RFC _requires_ Return-Path.  It is standard.  Envelope-To is something
that somebody made up.  I can't recall it being mentioned in any smtp rfc
that I've read recently.  I can't recall seeing it generated by any MTA.
You keep asserting that it has something to do with this and that I should
somehow "know" how it fits into the discussion.  I don't believe that
Envelope-To has anything to do with discussing whether or not _lmtpd
should make sure that its in-memory data matches what it writes to disk_.

> > On Tue, 23 Sep 2003, Pat Lashley wrote:
> >
> >> --On Monday, September 22, 2003 22:40:00 -0700 Chris Stromsoe
> >> <cbs at cts.ucla.edu> wrote:
> >>
> >> > That, imho, is not correct behavior.
> >>
> >> What headers, aside from Return-Path, are not cached in RAM?
> >
> > The Received header that lmtpd adds and the X-Sieve header.  There may be
> > others in other areas of the code, but those two and the Return-Path
> > header are the three that I saw.
>
> Well, it makes sense taht the X-Sieve header wouldn't be applied until
> sieve has processed the message.

The x-sieve header is applied regardless of sieve script processing.  If
you enable sieve with cyrus, it is _always_ added.

> As to lmtpd's Received header; I could certainly see an argument for
> including it if lmtpd has already sent back a positive response to the
> MTA.  However, if LMTP runs sieve while the LMTP connection is still
> open, then it is right to withold that header until it has actually
> accepted the message.

Two things:

 - lmtpd has written the received header to disk.  When that message is
forwarded or bounced the on disk file is sent.  If on-disk format is
canonical, then the header cache needs to have the headers too.  If the
header cache is canonical, then the on-disk representation needs to not
have those headers until the message is dispensed with.

 - sendmail provides a Return-Path header to procmail, the final delivery
agent.  sendmail could also provide one to lmtpd, but lmtpd provides its
own.  lmtpd is the "local mail transport protocol".  Like a local mailer
in sendmail.  It is the final delivery step.  Sure, other things can
happen after that; those things have nothing to do with anything that
happened up to that point.  When sieve forwards or rejects a message, it
invokes sendmail as a client.  It doesn't try to relay the message through
the next hop or anything like that.  It reinjects the message.

> > There is no RFC or draft that says that the return-path header should
> > not be visible as a header to sieve scripts, whereas 2821 does say
> > that the final delivery must add the return-path header.  To me, that
> > says that return-path should be added by lmtpd and be visible to
> > sieve.
>
> You are assuming that sieve processing occurs -AFTER- final delivery.
> The authors of lmtpd apparently consider it to be happening -BEFORE-
> final delivery; with the sieve script itself determining whether there
> will even BE a final delivery at that point.

Or, the authors of lmtpd never considered this situation and it wasn't a
big deal having or not having the missing headers in the cache.  I'll let
them speak for themselves.

> It makes no sense to you because you continue to believe that it is a
> shortcoming and not a correct behavour.  Stop trying to justify your
> 'solution' and try to take an objective look at the problem. You will
> find that it is solved admirably by the envelope extension.

I objectively believe that lmtpd is maintaining an on-disk structure that
does not match the structure that it is maintaining in-memory.  I
objectively believe that this is wrong.  I noticed this when I was looking
to find out why something I wanted to do wasn't working.  The two things
are _completely separate_.

> >> You're trying to change LMTP to provide a header that WILL NOT EXIST
> >> in other sieve environments.  It will not be at all obvious to the
> >> naive user that that header was inserted at the last minute rather
> >> than following the message from its inception.
> >
> > I still don't see what it is that you are trying to say.  You seem to
> > be assuming some sort of lmtp/sieve interdependance.  Please correct
> > me if I'm wrong.
>
> There is an interdependance -in this implementation- because lmtpd is
> the agent that invokes the sieve processing.  Your mistake is in
> considering the sieve processing to be done -after- lmtpd performs final
> delivery.  In actual fact, it is done once lmtpd has accepted the
> message but -before- it determines whether there will even be a final
> delivery.  It cannot know whether there will be a final at this point
> until the sieve script has been run.  Therefore it MUST NOT (in RFC
> terminology) add the Return-Path header before sieve processing.

lmtpd is the final delivery.  Read more of 2821:

   When the delivery SMTP server makes the "final delivery" of a
   message, it inserts a return-path line at the beginning of the mail
   data.  This use of return-path is required; mail systems MUST support
   it.  The return-path line preserves the information in the <reverse-
   path> from the MAIL command.  Here, final delivery means the message
   has left the SMTP environment.  Normally, this would mean it had been
   delivered to the destination user or an associated mail drop, but in
   some cases it may be further processed and transmitted by another
   mail system.

Which actually implies that lmtpd is wrong to be adding the Return-Path
and that the smtp server handing off to lmtpd should be adding it.  Which
is beside the point.

> > As far as being not obvious to the naive user ... when has that
> > stopped anything from happening?
>
> Wasn't the basis for your original 'problem' that naive users could not
> understand why they couldn't filter on the Return-Path header?

No.  I didn't say that anywhere.  Are you reading what I'm writing or just
looking to argue with me?

> >> You are confusing functionality with implementation.  A common mistake.
> >
> > No, I'm not:  I need the functionality of being able to filter on all of
> > the headers in the message.
>
> No, that's the implementation that you want, not the functionality that
> you need.  Sorting those two out is often one of the most difficult
> tasks for the software engineer.

Say again?  I know the functionality of what I want.  I don't care how it
is implemented.  There are headers.  I want to be able to filter on all of
them.  Not a subset of them, but all of them.  I'm not overly concerned
with how that is implemented, just that it is implemented.

> >                             Return-Path is a header.
>
> Once final delivery has actually been made, it is a header.  Until then
> all you have is the envelope sender.

final delivery occurs when smtp passes the message to lmtpd.  What lmtpd
does with it afterward is its own business.

> > If there were a separate sieve parser (a la procmail) and lmtpd forked
> > off sieve to take action, it would act on the on-disk file and would
> > see the Return-Path header.  The built-in sieve parser should see the
> > same set of headers.
>
> If there were a separate sieve parser, lmtpd would quite probably feed
> it the message (without a Return-Path header) via a pipe;

Look into how sendmail feeds procmail.  I use procmail to filter on
return-path all the time.

> and the seive
> processor would need to invoke some other mechanism to actually perform
> a final delivery.  If lmtpd did  use the file, it would quite probably
> NOT include the Return-Path header in that file.  (Because to do so
> would be an RFC violation since final delivery has not yet been made.)

sieve opens the on-disk file and _feeds it to sendmail_ for redirecting.
The on-disk file that _includes the Return-Path_.

So, is the on-disk file just an optimization?  Or is it the "final
delivery" and being used as such by other parts of lmtpd?

Have you looked at the code to see what it's doing?  Is there some
technical fault with what I'm proposing?  Or are you being disagreeable
because you don't personally like the return-path header or because I've
somehow offended your sensibilities?



-Chris