cyrus-imapd 2.1.15, sieve, lmtpd, and return-path header

[Chris Stromsoe]

On Mon, 22 Sep 2003, Pat Lashley wrote:

> Return-Path doesn't exist at all until final delivery, at which point, I
> believe it is optional.

from RFC2821 (which, I realize, is smtp and not lmtp)

   When the delivery SMTP server makes the "final delivery" of a
   message, it inserts a return-path line at the beginning of the mail
   data.  This use of return-path is required; mail systems MUST support
   it.  The return-path line preserves the information in the
   path from the MAIL command.


> And, as I recall, the Envelope-To: header is completely non-standard;
> though widely used; and when used, it is in a manner similar to Return-
> Path:  (I.e., Any existing one is removed and a new one is added only at
> the point of final delivery.)

I'm not familiar with Envelope-To and have no idea what uses it or might
use it.  I don't really care about (and am not complaining about) any
specific headers.

My problem is this:

  - a message is passed to lmtpd via lmtp
  - lmtpd writes the mail headers to disk and caches the headers in ram
  - lmtpd generates and writes a Return-Path header to disk but does not
    cache it in ram
  - lmtpd generates and writes other headers to disk but does not cache
    them in ram either
  - sieve operates by checking against the headers in the cache
  - sieve can't check all of the headers, because not all of them are in
    the cache

That, imho, is not correct behavior.

> What is your MTA?  Have you checked to see whether that header is
> optional? (Or does LMTPd create it?)  Perhaps you -shouldn't- see the
> Return-Path header when you pull a message from the store.

lmtpd generates a Return-Path for every message.  Look at lmtpengine.c
around line 1002.  The MTA isn't important as far as the return path is
concerned in this case:  it isn't the final delivery step, lmtpd is.

> --On Monday, September 22, 2003 19:32:07 -0700 Chris Stromsoe <cbs at cts.ucla.edu> wrote:
>
> > The expectation is that sieve will be able to match on any header in
> > the message.  But, right now, it can't, because the generated headers
> > are only written to disk and never inserted into the header cache.
>
> What, if anything, does the Sieve RFC have to say about Return-Path? I
> suspect that if it were allowed by the standard, then there would have
> been no need for an "envelope" extension.  (Does the envelope RFC/draft
> say why it is necessary?)

I don't what the Sieve RFC says about return-path.  I'm pretty sure that
it isn't germane to this topic.

> Smartsieve should either be smart enough to automatically translate the
> condition; or smart enough to warn the user that it won't work as
> expected.

smartsieve can't be smart enough to automatically translate the condition.
It seems like you're suggesting that smartsieve (and any other program
that generates sieve rules) should be following behind lmtpd, trying to
guess at what special header might map to which special sieve function.
I don't believe that is a sustainable long-term model.  lmtpd should make
sure that the on-disk headers actually match what is stored in the header
cache.

> > I'm not sure why the script won't work with any other conformant sieve
> > implementation.  This has nothing to do with sieve... it has to do with
> > lmtpd writing one thing to disk and keeping a separate thing in ram.
>
> Because other sieve implementations are likely to be in an environment
> where that header does not yet exist to be tested.

I'm not sure what you mean here.  Or maybe I am not being clear.  To
restate myself:  this issue has _nothing to do with sieve_.  I am not
trying to patch sieve and I don't want to be.  The patch changes the
behavior of lmtpd so that the on-disk and in-memory represention of the
headers is the same.  (Rather, the patch is a start; since it doesn't yet
include the last-hop Received header they don't completely match.)

> Your solution has the advantage of being simple and fairly easy to
> implement.  That doesn't make it the right one.
>
>
> The envelope extension exists to provide the functionality you need.

The envelope extension does not provide the functionality that I need.
Open any message that is stored in cyrus and was delivered via lmtp.
Look at the headers.  The first header that you will see looks something
like:

  Return-Path: <cbs at cts.ucla.edu>

I need to be able to filter on that header.  In my opinion, sieve code
like

  if anyof (header :contains "return-path" "cbs at cts.ucla.edu") {
     fileinto "cbs";
  }

needs to work



-Chris

> -Pat
>