need organizational hint

[David Lang]

Phil,
  I think I do understand what you are looking for and I think there is a
use for it. however one problem with having no secondary MX is that if
your machine is unreachable there are some #@!@#$ MTA's that will end up
sending the mail to the A record for your domain instead of waiting and
sending it to the MX record when it comes up. if you create domain names
exclusivly for e-mail this can be made to work correctly, but in most
cases today people want e-mail for domain.com to go to one server and HTTP
traffic for that domain to go to another. since DNS contains the MX record
to redirect the mail but there is not a nice way to redirect the web
traffic domain.com frequently ends up pointing to the same IP address as
www.domain.com. the result is MTA's that attempt to deliver mail to your
webservers.

for your job it sounds like a compile option for cyrus to run sieve
earlier in the receiving process and the corresponding options for
rejecting messages combined with a very small shim to be the MTA->LMTP
layer would do the job. for starting code for the MDA part I would suggest
looking into SMTP application proxies, since LMTP isn't designed to be
exposed to the world it's more likly that there are bugs lurking there
then in areas that are more exposed and the SMTP proxies do this sort of
checking (and the IP blacklisting you are also looking for) already. I
know that there are some proxies out there that do connect to the real
destination and pass the result code from it back to the sending machine
(the Raptor firewall proxy does this for example, I don't know what
opensource proxies also do this)

In my case I think I have just a bit to much processing that I want to do
with the incoming mail (in terms of address re-writing, etc) to be able to
use your approach, but when I learned about LMTP I had similar thoughts to
yours about being able to simplify the mail store machine by not running a
full MTA on it and just having the main MTA machine deliver the messages
to it. you are taking it a step farther by haaving the inbound mail bypass
your main MTA machine. if enough of your recived mail if from the outside
world instead of from your own network this may be a win (ignoring the
legal/etc reasons for not wanting to do delayed bounces)

David Lang

 On Sat, 12 Apr 2003, Phil Howard wrote:

> Date: Sat, 12 Apr 2003 23:10:30 -0500
> From: Phil Howard <phil-info-cyrus at ipal.net>
> To: Michael Bacon <baconm at duke.edu>
> Cc: info-cyrus at lists.andrew.cmu.edu
> Subject: Re: need organizational hint
>
> On Sat, Apr 12, 2003 at 02:11:46PM -0400, Michael Bacon wrote:
>
> | First, with regards to what you're generally trying to do, I think the
> | general consensus around here is that you're asking for trouble.  For most
> | of your goals, if you try to impliment them in the way that you're
> | considering, you're going to either spend a very long time writing a ton of
> | code that will bloat Cyrus beyond recognition, or you're going to end up
> | with your standards compliance in shreads.
>
> I have just another thing to add to this, which I thought of while out
> driving on some errands since posting my previous reply.
>
> The traditional approach I have seen in the past for the organization
> of a mail server involved a distinct separation between the mail store,
> which traditionally was the infamous mailbox format, and the access to
> that mail store, whether an MUA, or a daemon like POP3.  One program
> would deposit the mail in a standard form, and an entirely different
> one would fetch the mail in that same standard form.  These standard
> forms of delivery were generally integrated into the mail system, which
> of course has traditionally been sendmail plus a basic mail program.
>
> Before examining Cyrus, I was looking at using a Maildir format mail
> store, letting Postfix deposit mail, and using Courier IMAP to fetch it.
> It was suggested to me by a couple people to take a look at Cyrus for
> reasons that it had better ACL facilities, and performed better.
>
> What I found out was that Cyrus is rather monolithic in the sense that
> it is the same package which deposits mail and fetches it.  It is, as
> someone here pointed out, a "black box" with input (LMTP) and output
> (IMAP generally).
>
> So I could choose between:
>
> 1.  A monolith of MTA + MDA and a separate MAA (Mail Access Agent).
>     Postfix and Courier IMAP is what I had considered.
>
> 2.  A separate MTA and a monolith of MDA + MAA.  This would be Postfix
>     or something else, plus Cyrus.
>
> But what I see is that having a monolith is actually a good thing, not a
> bad thing, within reason.  The question, and probably the difference of
> opinion, will be just how far that should go.  In my view, Cyrus has
> already taken one step towards the monolithic mail server by combining
> the Mail Delivery Agent and the Mail Access Agent together.  That appears
> to me to have been done for a good reason, which is to enable a custom
> mail store architecture to enhance performance over things like a plain
> Maildir mail store (which lacks a store of metadata).
>
> It might have been possible to do that without creating a monolith.  But
> that would have meant creating yet another standard mail store format and
> all the associated documentation of it.  But by being a "black box", it
> gets the advantange of being able to change the design of that mail store
> at any time without the pressure of standards conformance.  Instead of
> standardizing on a format, or even an API, the standard went to a protocol
> (LMTP).
>
> What my idea does is carries that concept just one step further.
>
> There is an advantage to having the MDA and MAA integrated.  I believe
> there is also an advantage to having the MRA (Mail Receiving Agent, a
> subset of MTA functionality) integrated with the MDA.  I don't intent
> to carry that to being a full MTA integration ... just an MRA thing.
>
> The problem with this is that the advantages to MRA + MDA are not going
> to be realized everywhere.  In some environments (such as EDU), the
> issues are simply not the same as in some others.  And even in provider
> environments, not everyone is looking at all the cost factors, yet.
> There is still a lot of "unlimited mail for one fixed price" attitude
> while the people who take advantage of it and email several gigabytes
> per day get accounts cut off and peple who mail maybe one or two a week
> end up paying the same average price.  While the market isn't crying
> out for it, there is a business advantage to being able to provide a
> service to both ends of the usage spectrum ... as long as they don't
> have to pay for spam.  And there is the issue of being competetive in
> the market by making sure your services are more efficient by making
> sure your servers don't have to deal with so much mail your customers
> don't want to pay for.  These are things the EDU mail administrators
> never have to worry about.
>
> Completely monolithic mail server products already exist for Windows.
> I know people who are using them.  They do the whole job from SMTP to
> IMAP.  I don't know what their implementations actually look like, so
> they may be doing things like queueing internally.  I suspect as much
> since I do usually get an MDN instead of a DSN for things like an abuse
> mailbox not existing.  So their monolithic aspect may be more marketing
> than it is implementation.
>
> Back when this whole project was conceived, had I just jumped in and
> started to code from scratch, I'm sure that in the 2 years since I
> would by now have a working SMTP and IMAP monolithic server.  And I
> probably would have a custom mail store to make meta data lookups go
> a lot faster.  And I probably would be using Berkeley DB for that and
> separate files for content.  It might well be very similar to what
> Cyrus is today (although I would have done "virtual" domains from the
> beginning ... another concept business needs that the EDU environment
> typically does not unless they also do departmental stuff).
>
> My main reason for not wanting to queue mail is to avoid having to do
> any Message Disposition Notification.  The reasons for that include to
> make sure my server doesn't end sending mail that might be considered
> to be spam or similar abuse by someone whose email address was forged
> as the sender address by a spammer.  Further, my server could end up
> being blacklisted by places like AOL should some spammer try to send
> the same spam to thousands of my users; the mass MDN returns could be
> treated as a spam burst by AOL and others, triggering an automatic
> blacklist.  It's a shame that things have gotten to be that way, but
> that is a sad reality now.
>
> In many other cases, the MDN can't be delivered due to a non-responding
> MX host, and hence occupies my queue for a while.  That's a cost that
> gets passed to my pricing, so I'd rather avoid it.
>
> Almost all legitimate mail comes from an MTA that queues the mail it
> is trying to send.  So I see no reason I need to queue it.  If my mail
> server is down, I really don't need a secondary MX host to handle it.
> Most spamware delivered spam might get lost due to the outage.  That
> would be a plus.  So by shedding any need to queue mail, I gain back
> more control over the mail services, and can keep costs under control
> as well.
>
> --
> -----------------------------------------------------------------
> | Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
> | phil-nospam at ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
> -----------------------------------------------------------------
>