need organizational hint

[Phil Howard]

On Sat, Apr 12, 2003 at 02:11:46PM -0400, Michael Bacon wrote:

| Sorry to come in to this late -- I've been tied up all week.
| 
| First, with regards to what you're generally trying to do, I think the 
| general consensus around here is that you're asking for trouble.  For most 
| of your goals, if you try to impliment them in the way that you're 
| considering, you're going to either spend a very long time writing a ton of 
| code that will bloat Cyrus beyond recognition, or you're going to end up 
| with your standards compliance in shreads.

How about suggesting an alternative?  Putting all that in the MTA would
be as much, or more, bloat.


| I guess specifically I'm referring to the SMTP-directly-to-Cyrus bit. 
| Before you go calling this a doable task, you really should go back and 
| re-read RFC 2821.  You can't just include the SMTP functions you want to 
| include and expect to be free of trouble; the reason MTAs are as big and 
| complicated as they are is to handle all of the complexities of standards 
| compliance correctly.  You're not doing yourself a favor by trying to 
| replicate the functionality of one separately.

I'm not going to be doing a "whole MTA".  It will only be serving the
needs of receiving mail addressed to that specific mail store.  Anything
else is rejected.


| All of that said, the other reason not to go through all of this work is 
| that there is *almost* enough existing, tested software to do exactly what 
| you want to do, if you're willing to take the time to investigate.  I'll 
| outline two slightly different ways of going about this here -- they'll 
| both involve doing everything pretty much with Cyrus (of course) and 
| Sendmail, because that's what we use here and that's what I know the best. 
| It may be possible with one of the other MTAs as well -- I just don't know.

Sendmail has been ruled out a few years ago.  I've used Qmail and since
switched to Postfix.  I'm not committed to stay with Postfix, but it does
well for general MTA purposes, so I see no reason to abandon it for that
purpose.  But I don't have a problem with bypassing it for incoming mail.

There is still the option of a daemon designed to front-end a mail store.


| Strategy $1 -- This one is the first one I thought of reading this, and 
| would require a modification of the lmtpd code.  The key bit here is that 
| you run sendmail in interactive delivery mode (DeliveryMode=interactive in 
| your sendmail.cf).  This does basically what you described in an earlier 
| post -- sendmail makes the remote MTA hang on the line for the final return 
| code of the DATA command while it goes ahead and makes one delivery 
| attempt, which in this case is an LMTP connection to Cyrus.  If the 
| delivery succeeds, the remote MTA gets a 250 code back that the message is 
| accepted, and it never touches the queue.  If there's a temporary failure 
| in delivery, sendmail returns a 250 to the remote MTA and queues the 
| message locally for later retry.  If there's a permanent failure with 

No queueing.  Not even on a temporary failure.  The solution needs to send
back the actual failure code.  If Cyrus gives a temporary failure like 450,
then the SMTP daemon needs to pass 450 to the sending MTA.

Queueing is bad because it might be rejected later.  That then commits the
MTA that queued it to send a bounce message.  And I don't want that commitment
for several reasons, including legal ones.


| delivery (e.g., 550 User rejects message), I *think* that gets passed back 
| to the remote MTA, and the message is rejected.  I say I think because I 
| don't have a good way to test this at the moment.  This keeps in line with 
| standards compliance, because sendmail never actually returns the 250 
| message until it queues, so if sendmail crashes in the middle of the 
| transfer, the remote MTA never (better not!) dequeues the message, and will 
| resend when the system recovers.  This is a bit of an oversimplification of 
| the process, but hopefully you get the idea.

As far as I know, sending the 450 back over SMTP when it gets one via LMTP
would not violate RFC 2821.


| Now, as far as the user-configurable stuff, you can theoretically do pretty 
| much everything you need with SIEVE.  You earlier implied that sieve can't 

Apparently that is not the case.  Based on a review of the language in RFc
3028, this is a tool just for content filtering.  That has uses and Sieve
appears to be a good way to do it.  But that's not all that will be done.
There will be filtering done BEFORE receiving the DATA command, too.


| reject a message, which is only partially true.  The "reject" action in 
| sieve does exactly what you would want it to on a large scale basis.  On 
| the other hand, and this is where the modification would need to be done, 
| lmtpd currently doesn't do what you want it to when it encounters a sieve 
| reject.  LMTP accepts the message provided the user exists, is under quota, 
| and that sort of thing, then later processes it with sieve.  In the case of 

That is something that needs to be changed for messages that need to have a
rejection sent back to the sender.  If the user prefers their rejections to
be silently discarded (seemingly in violation of the RFC), or diverted to
an alternate folder, then filtering later is fine.


| a reject, it does not write it to the mailbox, but instead makes an 
| external call to sendmail to send a rejection in the MDN notification 

And this is specifically not acceptable for my goals for several reasons.
I want all rejections to be in the form of 5XX codes during SMTP.


| format.  Theoretically (and I haven't spent enough time with the lmtpd code 
| to know how possible this is), you could move processing of the sieve 
| script up in the chain, so that if a reject were encountered, you could 
| send back a 5xx code to sendmail as the recipient-specific return code, 
| which would (in interactive mode) get passed back to the remote MTA, and 
| the message would be rejected.  Now, you'd be in violation of the SIEVE 
| RFC, because you'd be sending back a DSN instead of an MDN, but I think you 
| could live with that.  Of course, the code probably wouldn't get accepted 
| into the cyrus distro, but if you're okay with being a lone wolf that way, 
| then it doesn't matter much.

I'm committed to not ever sending any MDN.  So I guess I have to live with
being in violation of RFC 3028.


| How does that information get passed to the users?  You convince sendmail 
| to stamp all of the interesting information you want into the headers of 
| the message, then get Sieve to filter on that instead.  You do this via the 
| milter interface, which I get into more detail with below.
| 
| Anyway, that's the strategy that's closest to what you were describing. 
| For a good bit more coding work, but neater coding work and more standards 
| compliance, you might consider...
| 
| 
| Straegy $2 -- You leave Cyrus alone, run sendmail in interactive mode, and 
| write yourself a nice fancy milter to do everything you want to do.  I 
| can't describe the milter interface in its entirety here, but effectively 
| it's an API that allows you to run a threaded process external to the MTA 
| that does the interesting filtering for you.  If done correctly, it has 
| access to any and all information that sendmail has that you might want to 
| filter on, including the envelope, the message headers, the message body, 
| and connection information (such as the remote IP and such).  Filtering 
| happens as the message is submitted, and the milter can halt the process at 
| any point and reject or discard the message.  Furthermore, at the end of 
| message submission, the milter may add headers, change headers, or replace 
| the body of the message.  In short, you can filter mail about any way you 
| could imagine with it, and you do it in-band upon message submission, so 
| you can reject the thing if you want to.
| 
| So, if you really want to write in a bunch of user-specific stuff and 
| manage the database yourself, you could just write all of that into a 
| milter and run that alongside sendmail.  As a teaser, here at Duke, we've 
| just written a generic regex-based milter that will hopefully take the 
| place of procmail in our delivery setup, saving us tons of CPU cycles and 
| giving performance a nice kick in the pants.  (It's still in the final 
| stages of testing -- we'll hopefully be releasing it sometime soon.)  It 
| doesn't do user-specific filters, but there's no reason you couldn't do 
| something similar and write that in.

I won't be using sendmail, and certainly not trying to implement a bunch
of stuff on it.  A whole new daemon dedicated to doing the "interactive"
mode exclusively, with filtering, is an option.  But the question is how
to get Sieve to filter at DATA time rather than later.


| If you've really got the jones to code something, I'd go with #2.  Rather 
| than re-inventing the wheel, you should be able to do everything you want 
| with a little glue code, and still take advantage of the best of existing 
| projects.  There may be a way to do something similar in Postfix, if you're 
| really set on using that, but I'm not the one to help you there.

I'm certainly big on coding, but definitely not with anything to do with
sendmail.  C is my big language.  So the front-end is one idea, and an
added feature to Cyrus is another.  In the latter case it can be a drop in
feature which is quite common for lots of other packages anyway.


| But I have to agree with all of the other sentiments expressed here. 
| Wedging SMTP into Cyrus is a Bad Idea (tm).

The reason the idea is considered is because I see no reason for having
all those queueing and transfer steps involved just to get mail from an
SMTP session into a mail store ... aside from standards which have not
kept up to date with current technical and legal issues (especially spam).
I feel that a mail server needs to be at least logically monolithic in
terms of functionality (for example being able to do the DSN reject for
all reasons of rejection ... if that is the local policy or user preference).
Maybe it doesn't have to be monolithic in actual implementation, but if
I were to build such a server from scratch, it would be more monolithic,
which would take mail directly from the incoming SMTP session and put it
directly in the mail store ... no queueing and no pickups ... just direct.
The thing is, such a monolithic design would be only barely larger than
what Cyrus is now by expanding on LMTPd to be SMTPd, and adding the meta
filtering capability, and moving Sieve to real time.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam at ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------