cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Pat Lashley]

--On Tuesday, September 23, 2003 02:18:50 -0700 Chris Stromsoe 
<cbs at cts.ucla.edu> wrote:

> On Tue, 23 Sep 2003, Pat Lashley wrote:
>
>> --On Monday, September 22, 2003 22:40:00 -0700 Chris Stromsoe
>> <cbs at cts.ucla.edu> wrote:
>>
>> I keep mentioning Envelope-To because 30+ years of Software Engineering
>> experience tells me that if you handle Return-Path, people will also
>> want Envelope-To.
>
> out of ~40,000 messages I've got handy, between 50 and 60 had some variant
> of envelope-to (either plain or with x- or old-), and 36,906 had a
> return-path header.  Return-Path is an rfc mandated header that must be
> handled.

None of the messages should have had either of those headers when
you recieved them.

Yes, the RFC mandates a Return-Path header; but it specificly says
that it is to be generated only at the point of final delivery into
a file.

>        I'm still not sure how Envelope-To applies to this discussion.

Please re-read the first quoted paragraph above.  Repeat until you
DO understand how Envelope-To applies to this discussion...


>> > That, imho, is not correct behavior.
>>
>> What headers, aside from Return-Path, are not cached in RAM?
>
> The Received header that lmtpd adds and the X-Sieve header.  There may be
> others in other areas of the code, but those two and the Return-Path
> header are the three that I saw.

Well, it makes sense taht the X-Sieve header wouldn't be applied until
sieve has processed the message.

As to lmtpd's Received header; I could certainly see an argument for
including it if lmtpd has already sent back a positive response to the
MTA.  However, if LMTP runs sieve while the LMTP connection is still
open, then it is right to withold that header until it has actually
accepted the message.

>> > I don't what the Sieve RFC says about return-path.  I'm pretty sure
>> > that it isn't germane to this topic.
>>
>> Then why was it necessary to create an "envelope" extension to sieve?
>
> I don't know.  Maybe early versions of lmtpd weren't following rfc2821 and
> properly settings the return-path, and sieve couldn't get at the
> information any other way.  Maybe the envelope extension was added to get
> at envelope-to and envelope-from was added as an afterthought.

Or, maybe, it is just more correct to think of and process the
envelope as separate from the in-message headers.

>> Nonsense.  I'm only suggesting that it be smart enough to actually
>> follow the RFCs and drafts; and to know about a few of the extensions
>> like envelope.
>
> There is no RFC or draft that says that the return-path header should not
> be visible as a header to sieve scripts, whereas 2821 does say that the
> final delivery must add the return-path header.  To me, that says that
> return-path should be added by lmtpd and be visible to sieve.

You are assuming that sieve processing occurs -AFTER- final delivery.
The authors of lmtpd apparently consider it to be happening -BEFORE-
final delivery; with the sieve script itself determining whether there
will even BE a final delivery at that point.

And you should also remember that there's nothing about sieve that
restricts it to delivery-time or post-delivery-time.  It could just
as easily be integrated into an MTA for relay-time processing; or
into an MUA for use during the send.

>                                                                Making
> smartsieve -- and every other application out there that generates sieve
> scripts -- work around lmtpd's shortcomings doesn't make any sense to me.
> Fixing lmtpd so that the in-ram cache mirrors the on-disk data store does.

It makes no sense to you because you continue to believe that it is
a shortcoming and not a correct behavour.  Stop trying to justify
your 'solution' and try to take an objective look at the problem.
You will find that it is solved admirably by the envelope extension.


>> You're trying to change LMTP to provide a header that WILL NOT EXIST in
>> other sieve environments.  It will not be at all obvious to the naive
>> user that that header was inserted at the last minute rather than
>> following the message from its inception.
>
> I still don't see what it is that you are trying to say.  You seem to be
> assuming some sort of lmtp/sieve interdependance.  Please correct me if
> I'm wrong.

There is an interdependance -in this implementation- because lmtpd
is the agent that invokes the sieve processing.  Your mistake is in
considering the sieve processing to be done -after- lmtpd performs
final delivery.  In actual fact, it is done once lmtpd has accepted
the message but -before- it determines whether there will even be
a final delivery.  It cannot know whether there will be a final
at this point until the sieve script has been run.  Therefore it
MUST NOT (in RFC terminology) add the Return-Path header before
sieve processing.


> As far as being not obvious to the naive user ... when has that stopped
> anything from happening?

Wasn't the basis for your original 'problem' that naive users could
not understand why they couldn't filter on the Return-Path header?


>> You are confusing functionality with implementation.  A common mistake.
>
> No, I'm not:  I need the functionality of being able to filter on all of
> the headers in the message.

No, that's the implementation that you want, not the functionality
that you need.  Sorting those two out is often one of the most
difficult tasks for the software engineer.

>                             Return-Path is a header.

Once final delivery has actually been made, it is a header.  Until
then all you have is the envelope sender.

>                                                      lmtpd is the "final
> delivery"  outside of the smtp environment.  lmtpd adds the header on
> disk.  It does not cache the header in the header cache.

That's an implementation detail that should be opaque to you.  An
optimization based on the likelyhood that the sieve script will decide
that a final delivery should be made.


> If there were a separate sieve parser (a la procmail) and lmtpd forked off
> sieve to take action, it would act on the on-disk file and would see the
> Return-Path header.  The built-in sieve parser should see the same set of
> headers.

If there were a separate sieve parser, lmtpd would quite probably feed
it the message (without a Return-Path header) via a pipe; and the seive
processor would need to invoke some other mechanism to actually perform
a final delivery.  If lmtpd did  use the file, it would quite probably
NOT include the Return-Path header in that file.  (Because to do so
would be an RFC violation since final delivery has not yet been made.)




-Pat