cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Pat Lashley]

--On Tuesday, September 23, 2003 17:55:42 -0700 Chris Stromsoe 
<cbs at cts.ucla.edu> wrote:

> My measurements were not scientific.  I'm sure that I overestimated the
> count of messages by some factor.  My point was that envelope-to does not
> seem to be in wide use.

All you've managed to prove is that you don't use it.  As I said
before, sites that -do- use it add it at the last minute before
final delivery (like Return-Path).  And most sites that use it
immediately remove any existing Envelope-To: header on incoming
messages.


> Any smtp server that removes a header from a message that it is relaying
> (I'm assuming that you mean relaying when you say "passing on") is (I'm
> fairly certain) violating rfc2821.  Only the end points should be removing
> headers.  I've never worked on a system that added the envelope-to header.

Actually, I suspect you'll find that removing Return-Path on incomming
messages is at least recommended; if not required.  Since Envelope-To
is non-standard; there is no standard for handling it.  As I have said
several times now, sites that do use it, use it in a manner analagous
to the Return-Path; which dictates removing any existing version on
incomming messages and only generating one for final delivery.


> LMTP is not a dialect of SMTP (unless I'm reading 2033 wrong -- feel free
> to correct me if I am):
>
>    Although LMTP is an alternative protocol to ESMTP, it uses (with a
>    few changes) the syntax and semantics of ESMTP.  This design permits
>    LMTP to utilize the extensions defined for ESMTP.  LMTP should be
>    used only by specific prior arrangement and configuration, and it
>    MUST NOT be used on TCP port 25.

I would certainly interpret "uses ... the syntax and semantics of
ESMTP" as indicating that it is, in effect, a dialect of SMTP.  What
part of that paragraph leads you to disagree?


>> My actual assertion is that users, given the ability to filter on the
>> envelope sender, will also want to filter on the envelope recipient. If
>> you use Return-Path as the handle with which to check the former; you
>> must consider Envelope-To as the appropriate handle for checking the
>> latter.
>
> Then write up an rfc modifying whatever the current SMTP definition is
> stating that mta's MUST handle the envelope-to header.

Are you intentionally missing my point; or am I just not being
sufficiently clear?  Let me try it again.  You want to provide
the ability to filter on the envelope sender.  Given that ability,
users will want to filter on on the envelope recipient.  They
will expect to be able to do the later in a manner analagous to
the way they do the former.  That implies the use of a header
and normal header conditions.  Envelope-To is the de-facto
standard for storing the envelope recipient in the message.

There is no need for a modification or extension to SMTP because
NEITHER Envelope-To nor Return-Path headers should be used for
sieve filtering.  That's what the envelope extension is for.


Oh, and as to how widespread the use of Envelope-To is, there
is at least one widely used MTA (exim) which provides a simple
flag to turn it on; and one to remove existing Envelope-To:
headers on incoming messages.  I suspect a sizable percentage
of Exim users are using it.  (But you don't see that because
they are using it correctly; so the header never escapes their
site.)  It would not surprise me to discover that other popular
MTAs have easy ways to add that header.  (Other than a generic
mechanism for adding any header at all.)


>                                                          Until then, it's
> not a big deal in my books.  You should also note that I never said that I
> wanted to be able to filter on the envelope sender or recipient.  I want
> to be able to filter on the complete set of headers as they will appear in
> my mailbox.  Regardless of what the ultimate data source is.

So if your MUA adds a header, you think that Sieve should somehow
be prescient enough to anticipate that and allow you to filter on
it?


> I don't care that return-path provides the envelope values.  I care that
> it is a header that is a part of the message that is on disk, but that is
> not a part of the header cache.

Please forget about the details of lmtpd's implementation.  That
the header has been placed in the disk copy in anticipation of
the likelyhood of final delivery does not change the fact that
conceptually, and following the proper strictures of the RFCs,
it does not yet exist when Sieve examines the message.

You don't care that the value comes from the envelope because you
have become married to the idea that "it's a header visible in my
mailbox and should be filterable".  You are ignoring the proper
timeline.  You are ignoring the presence of a method for filtering
on that data.  You are intent on having -your- solution whether
it is the right one or not.


>> But presumably, conceptually at least, -after- sieve has had a whack at
>> the message.  Note that the absence of a sieve script for a given
>> local-part does not mean that sieve doesn't process the message; only
>> that it uses the default action.
>
> No.  It is added regardless of whether or not sieve ever looks at the
> message.  If you have no sieve script, sieve is never invoked.  Yet
> x-sieve is still added.

That's another optimization.  Conceptually, sieve is invoked and
performs the default action; which is to file the message into
your INBOX.

(And if the lmtpd authors want to contradict me here; I will claim
that messages which are not examined by Sieve should not have an
X-Sieve header added.)


>> As such, it is a specialized dialect of SMTP; and is the proper agent
>> for adding the Return-Path header to messages which it actually delivers
>> into a local mailbox.  But -NOT- to messages which are bounced or
>> forwarded.
>
> lmtp is not a specialized dialect of smtp.  It is its own separate
> transport protocol that just happens to share some of the language of
> smtp.  smtp passing off a message to an lmtp server is semantically the
> same as smtp passing it off via ftp or nntp or whatever else.  From the
> perspecitve of the mta, it is the functional endpoint.

I think your very quote from the RFC above estabilshes that it -IS-
a dialect of SMTP.  One which is intended only for local transfers
rather than cross-domain transfers.  It's "Local Mail Transfer Protocol";
not "Final Delivery Mail Transfer Protocol".  From the perspective of
the MTA it is just a simplified way of passing the message off to
another -local- server for further processing.


>> I wish they would.  But I strongly suspect that they had a much more
>> global overview of mail processing than you are espousing.
>
> In my experience, which isn't as vast as yours, large scale projects in an
> academic environment rarely work that way (with a global overview).
> There are often times general goals expressed at the start of the project,
> time passes, the project becomes useful, other people see the use and
> start to contribute, things get passed around, and the project changes
> from its original use.

Hmm, I don't have the old messages; but weren't we talking about the
authors of the LMTP protocol/RFCs at this point; not the authors of
the Cyrus lmtpd?  The folks who write and review RFCs tend to take
a more global view than most implementers.


>> Wasn't working -as you expected it to-.  You seem to be completely
>> resistant to the idea that your expectations were wrong.
>
> You are correct.  I am completely resistant to the notion that my
> expectation in this situation is wrong.  sieve should have full access to
> all of the available headers.

It does.  The Return-Path header is not available at this point.
You are confused by the fact that a predictive copy happens to
appear in a file created by the same daemon.  But it is, quite
properly, -not- available to Sieve because the message has not
yet been designated for final delivery.

The headers that are available are the ones that are in memory.
All of which -are- accessable by Sieve.


> Return-Path is a header.  lmtpd knows about the header.  It should provide
> that header to sieve with all of the other headers it knows about.

But that header isn't in the incoming message; and doesn't properly
exist until after sieve is done with it.  At this point it is only
a potential header; not an actual one.  Should lmtpd provide -all-
potential headers for Sieve to process?  (If I had time, I'd go
looking for the RFCs and drafts relating to Message Delivery Notification.
I suspect there's a good reducto ad absurdum example in there somewhere.)


> That is not the case.  The functionality that I want is to filter on the
> headers as they will exist on disk.  The same as if sieve were called by
> forking "/usr/local/bin/sieve message-file" would.  The envelope extension
> lets you filter based on the envelope as provided to the lmtp server.
> That's fine.  I want sieve to be able to filter based on the headers.

Typical user.  Focused on your proposed solution to the point of
completely rejecting cleaner, more capable alternatives.

Once again.  The fact that that header is predictively inserted into
the disk file is immaterial.  It is even arguably wrong.  (Although
that argument is likely to fail in the face of the performance advantage
of not having to rewrite the file just to add that one header when a
final delivery is actually made.)

Despite a complete lack of evidence to support it, you keep assuming
that if sieve were to be run as a separate process, that it would be
given the message file as a parameter rather than using some simple
protocol over a pipe.  If it were done that way, it would almost
certainly take additional parameters to specify the envelope parameters.
(Or at least any version that implemented the envelope extension would.)


>> That's where we disagree.  I claim that final delivery doesn't occur
>> until it is actually placed into a mailbox.  Apparently the authors of
>> the lmtpd code tended more towards my view than yours.
>
> Hrm.  Very well.  I claim that when sieve is being used, final delivery is
> done by sieve.  When it is not in use, final delivery is done by lmtpd.
> Ie, one of
>
>   smtp -> lmtp -> mailbox
>   smtp -> lmtp -> sieve -> mailbox
>
> and not
>
>   smtp -> lmtp -> sieve -> lmtp -> mailbox

And I claim that final delivery is always

	smtp -> lmtp -> sieve -> mailbox

but that the 'sieve' step is optimized to a no-op in the case where
there is no sieve script.


>> It's been quite a while since I've used procmail; but I believe that is
>> a shortcomming of it's implementation.  Does it have any other mechanism
>> to get ahold of the envelope values?
>
> It may, I don't know (I've never looked).  In any event, procmail doesn't
> have to take the return-path -- my mailer that calls procmail inserts a
> return-path and feeds the message to procmail, which filters and delivers
> my mail.  It is (as far as I'm aware without doing a lot more reading than
> I have time for right now) 100% normal operating behavior for sendmail
> using procmail as a delivery agent.

Procmail is basicly a hack to provide much greater capability to the
unix .forward files.  Which were themselves basicly a hack.  The MTA
treats it as a final delivery, whether it actually is or not.  But
procmail is not well integrated into the mail system; and is most
emphaticly -not- a good model for a stand-alone sieve filter.

Compare against Exim filters; also available in the .forward file.
Exim handles them directly; so that various variables are available;
including the envelope values.  This is possible because it -is- a
well-integrated filtering system; much like sieve integrated with
lmtpd.  Sieve and Exim filters are both well-designed solutions
that fit smoothly into the mail processing environment.  Procmail
basicly sits outside the primary mail processing environment.

(Please don't take any of this as denigration of the utility of
procmail.  And it does have the major advantage of being usable
without any special support from the MTA.)



-Pat