cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Chris Stromsoe]

On Tue, 23 Sep 2003, Pat Lashley wrote:

> It wasn't clear whether you were testing incoming messages or ones
> already in mailboxes.  And your statement above certainly implies that a
> bit less than 10% of the messages do NOT have a Return-Path header.  Is
> that in fact the case; or was that 40,000 an overestimate?

grep ^From\  mail/* | wc -l
grep ^Return-Path:\  mail/* | wc -l
grep -i envelope-to mail/* | wc -l

My measurements were not scientific.  I'm sure that I overestimated the
count of messages by some factor.  My point was that envelope-to does not
seem to be in wide use.


> Systems that do generate Envelope-To headers generally only do so when
> they believe that the message is about to achieve final delivery; and
> generally remove Envelope-To headers that already existed on incoming
> messages, whether or not they will be passed on or delivered locally.
> (On the theory that a pre-existing header may not be trustworthy.)

Any smtp server that removes a header from a message that it is relaying
(I'm assuming that you mean relaying when you say "passing on") is (I'm
fairly certain) violating rfc2821.  Only the end points should be removing
headers.  I've never worked on a system that added the envelope-to header.


> LMTP is a dialect of SMTP; so it hasn't left the SMTP environment until
> sieve is through with it and lmtpd puts it into a cyrus mailbox. (If, in
> fact, that is what sieve says is to happen to the message.)

LMTP is not a dialect of SMTP (unless I'm reading 2033 wrong -- feel free
to correct me if I am):

   Although LMTP is an alternative protocol to ESMTP, it uses (with a
   few changes) the syntax and semantics of ESMTP.  This design permits
   LMTP to utilize the extensions defined for ESMTP.  LMTP should be
   used only by specific prior arrangement and configuration, and it
   MUST NOT be used on TCP port 25.


> > The RFC _requires_ Return-Path.  It is standard.  Envelope-To is
> > something that somebody made up.  I can't recall it being mentioned in
> > any smtp rfc that I've read recently.  I can't recall seeing it
> > generated by any MTA. You keep asserting that it has something to do
> > with this and that I should somehow "know" how it fits into the
> > discussion.  I don't believe that Envelope-To has anything to do with
> > discussing whether or not _lmtpd should make sure that its in-memory
> > data matches what it writes to disk_.
>
> My actual assertion is that users, given the ability to filter on the
> envelope sender, will also want to filter on the envelope recipient. If
> you use Return-Path as the handle with which to check the former; you
> must consider Envelope-To as the appropriate handle for checking the
> latter.

Then write up an rfc modifying whatever the current SMTP definition is
stating that mta's MUST handle the envelope-to header.  Until then, it's
not a big deal in my books.  You should also note that I never said that I
wanted to be able to filter on the envelope sender or recipient.  I want
to be able to filter on the complete set of headers as they will appear in
my mailbox.  Regardless of what the ultimate data source is.


> I also assert that it is wrong to use those headers as the mechanism for
> obtaining envelope values; and that the envelope extension to sieve
> provides the right mechanism.

I don't care that return-path provides the envelope values.  I care that
it is a header that is a part of the message that is on disk, but that is
not a part of the header cache.


> > The x-sieve header is applied regardless of sieve script processing.
> > If you enable sieve with cyrus, it is _always_ added.
>
> But presumably, conceptually at least, -after- sieve has had a whack at
> the message.  Note that the absence of a sieve script for a given
> local-part does not mean that sieve doesn't process the message; only
> that it uses the default action.

No.  It is added regardless of whether or not sieve ever looks at the
message.  If you have no sieve script, sieve is never invoked.  Yet
x-sieve is still added.


> >  - sendmail provides a Return-Path header to procmail, the final
> > delivery agent.  sendmail could also provide one to lmtpd, but lmtpd
> > provides its own.  lmtpd is the "local mail transport protocol".
> > Like a local mailer in sendmail.  It is the final delivery step.
>
> As such, it is a specialized dialect of SMTP; and is the proper agent
> for adding the Return-Path header to messages which it actually delivers
> into a local mailbox.  But -NOT- to messages which are bounced or
> forwarded.

lmtp is not a specialized dialect of smtp.  It is its own separate
transport protocol that just happens to share some of the language of
smtp.  smtp passing off a message to an lmtp server is semantically the
same as smtp passing it off via ftp or nntp or whatever else.  From the
perspecitve of the mta, it is the functional endpoint.


> I wish they would.  But I strongly suspect that they had a much more
> global overview of mail processing than you are espousing.

In my experience, which isn't as vast as yours, large scale projects in an
academic environment rarely work that way (with a global overview).
There are often times general goals expressed at the start of the project,
time passes, the project becomes useful, other people see the use and
start to contribute, things get passed around, and the project changes
from its original use.


> > I noticed this when I was looking to find out why something I wanted
> > to do wasn't working.
>
> Wasn't working -as you expected it to-.  You seem to be completely
> resistant to the idea that your expectations were wrong.

You are correct.  I am completely resistant to the notion that my
expectation in this situation is wrong.  sieve should have full access to
all of the available headers.


> >> Wasn't the basis for your original 'problem' that naive users could
> >> not understand why they couldn't filter on the Return-Path header?
> >
> > No.  I didn't say that anywhere.  Are you reading what I'm writing or
> > just looking to argue with me?
>
> Perhaps I misunderstood your stated reasons for wanting to provide the
> ability to filter on Return-Path instead of using the envelope
> extension.

Return-Path is a header.  lmtpd knows about the header.  It should provide
that header to sieve with all of the other headers it knows about.


> Again, you are focused too tightly on a particular symptom and ignoring
> the bigger 'problem'.  The functionality you want is to filter on the
> envelope sender.  The implementation that you want is to be able to use
> the Return-Path header.  But the envelope extension provides another,
> cleaner, implementation of the actual functionality that you desire.

That is not the case.  The functionality that I want is to filter on the
headers as they will exist on disk.  The same as if sieve were called by
forking "/usr/local/bin/sieve message-file" would.  The envelope extension
lets you filter based on the envelope as provided to the lmtp server.
That's fine.  I want sieve to be able to filter based on the headers.


> You can filter on all of the headers that actually exist at that point.
> The Return-Path header does -not- really exist yet.  That it appears to
> exist is an implementation detail that you shouldn't be aware of.  (I
> realise that this point is a but subtle.)

I disagree.  I believe that the reason that the generated headers are not
in the header cache is because they were overlooked, not intentionally
left out.  My patch addresses that.


> > final delivery occurs when smtp passes the message to lmtpd.  What
> > lmtpd does with it afterward is its own business.
>
> That's where we disagree.  I claim that final delivery doesn't occur
> until it is actually placed into a mailbox.  Apparently the authors of
> the lmtpd code tended more towards my view than yours.

Hrm.  Very well.  I claim that when sieve is being used, final delivery is
done by sieve.  When it is not in use, final delivery is done by lmtpd.
Ie, one of

  smtp -> lmtp -> mailbox
  smtp -> lmtp -> sieve -> mailbox

and not

  smtp -> lmtp -> sieve -> lmtp -> mailbox


> >> If there were a separate sieve parser, lmtpd would quite probably
> >> feed it the message (without a Return-Path header) via a pipe;
> >
> > Look into how sendmail feeds procmail.  I use procmail to filter on
> > return-path all the time.
>
> It's been quite a while since I've used procmail; but I believe that is
> a shortcomming of it's implementation.  Does it have any other mechanism
> to get ahold of the envelope values?

It may, I don't know (I've never looked).  In any event, procmail doesn't
have to take the return-path -- my mailer that calls procmail inserts a
return-path and feeds the message to procmail, which filters and delivers
my mail.  It is (as far as I'm aware without doing a lot more reading than
I have time for right now) 100% normal operating behavior for sendmail
using procmail as a delivery agent.




-Chris