cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Pat Lashley]

--On Tuesday, September 23, 2003 21:58:03 -0700 Chris Stromsoe 
<cbs at cts.ucla.edu> wrote:

> On Tue, 23 Sep 2003, Pat Lashley wrote:
>
> Just because lmtp is linked to sieve is no reason to assume that they are
> (or should be)  functionally intertwined.

I never claimed that they should be.  In fact, your arguments are
based much more on the current intertwined implementation than on
the conceptual sequence of events.

>                                           smtp does stuff and hands off
> to lmtp.  lmtp does stuff (figures out if the mailbox is local, adds
> return-path, received header) and hands off to sieve.

But it can't add the return-path header because final delivery is not
yet being made.  If, in fact, it handed off to sieve and then was done,
it should add the Received header and return a 200 code to the MTA.
BUT since LMTP doesn't do queuing; there's a strong argument for the
current behavour of running sieve before returning the 200 code. And
it has not, in fact, received the message until it returns the 200
code; so it should not yet add the Received header.

>                                                       sieve adds x-sieve
> header, filters, and does some action.

If the action includes filing into a mailbox, it adds the Return-Path
header.  (Or, in an intertwined implementation, whichever bit of code
actually stores it into the mailbox should be in charge of adding that
header.)

>                                         It would be nice if filtering were
> completely pluggable and could be replaced by other filtering agents, much
> the same way that you can plug different sasl libraries.

Sounds reasonable; assuming a suitably generic filtering library API.
Note that since it is an API, the data would almost certainly be passed
in memory, not in temp files...  (Although specific implementations
may use a temp file to communicate with an external process.)


>> You don't care that the value comes from the envelope because you have
>> become married to the idea that "it's a header visible in my mailbox and
>> should be filterable".  You are ignoring the proper timeline.  You are
>> ignoring the presence of a method for filtering on that data.  You are
>> intent on having -your- solution whether it is the right one or not.
>
> No, I believe that my solution _is_ the right one.

I have come to believe that the most powerful motivating force in
human affairs is the need to Be Right.  In most of us this unfortunately
manifests as the need to justify our current position instead of to
step back and obtain the necessary facts to determine what actually
-is- right.

>                                                     Headers added by lmtp
> should be visible to sieve.  Things shouldn't be done to the message after
> sieve processes it.

Those headers should be (conceptually at least) added as part of the
sieve processing.  Sieve processing occurrs -before- final delivery
is determined.  One of the sieve actions is to perform final delivery.
The Return-Path header should not be added until final delivery is
determined.  Therefore it should not be added until sieve indicates
that the message is to be placed into a mailbox.  The Received header
should not be added until lmtpd has actually indicated successful
reception of the message; which occurs after sieve is done.

Conceptually at least, those headers are -not- added by lmtpd; they
are added by sieve.  The intertwined implementation; along with some
obvious optimizations lead to the illusion that they are created by
lmtpd.

It could certainly be argued that the X-Sieve header should be added
by sieve before it does any further processing; but I really don't
see much application for filtering on that value.


>> But that header isn't in the incoming message; and doesn't properly
>> exist until after sieve is done with it.
>
> until lmtpd is done with it.  At which point sieve takes over and should
> have access to it.

But sieve takes over before lmtpd is done.  Sieve must finish successfully
for lmtpd to indicate successful reception of the message.  If lmtpd DID
accept the message before passing it off to sieve; then it should add the
Received header first.  But in that case, it should also not add the
Return-Path header at all; because that would be sieve's responsability.


>> At this point it is only a
>> potential header; not an actual one.  Should lmtpd provide -all-
>> potential headers for Sieve to process?
>
> lmtpd should provide all of the headers that it creates plus all of the
> headers that come with a message.

All of the headers which it has properly created before sieve processing
starts.  But the headers in question are not properly created until sieve
is done.

Postulate the addition of an 'X-Sieve-Cost:' header which holds some
information about processing time, memory usage, or other resource
consumption by the sieve process itself.  That would certainly be a
useful header to anyone concerned with sieve overhead or optimizing
scripts.  It would, of necessity, be added by the lmtpd.  But it
could not possibly be added before sieve processing began.  Yet your
'all headers that it creates' argument claims that it should be.


>> And I claim that final delivery is always
>>
>> 	smtp -> lmtp -> sieve -> mailbox
>>
>> but that the 'sieve' step is optimized to a no-op in the case where
>> there is no sieve script.
>
> If that is the case, sieve should see the full set of headers as generated
> by lmtpd.  Yet, it does not.  My patch addresses that.

No, no, no.  Those headers are, conceptually, according to the diagram
above, created by -sieve-, not lmtpd.  Therefore they should -not- be
visible to the sieve script.  In the sequence

	smtp --> lmtp --> sieve --> lmtp --> mailbox
	                                 \-> bounce/forward

then they are added by the -second- lmtp step.  The fact that both
lmtp steps and the sieve step are embodied by a single lmtpd process
is an implementation detail; not a fundamental constraint.



-Pat