need organizational hint

[Phil Howard]

On Sat, Apr 12, 2003 at 02:11:46PM -0400, Michael Bacon wrote:

| First, with regards to what you're generally trying to do, I think the 
| general consensus around here is that you're asking for trouble.  For most 
| of your goals, if you try to impliment them in the way that you're 
| considering, you're going to either spend a very long time writing a ton of 
| code that will bloat Cyrus beyond recognition, or you're going to end up 
| with your standards compliance in shreads.

I have just another thing to add to this, which I thought of while out
driving on some errands since posting my previous reply.

The traditional approach I have seen in the past for the organization
of a mail server involved a distinct separation between the mail store,
which traditionally was the infamous mailbox format, and the access to
that mail store, whether an MUA, or a daemon like POP3.  One program
would deposit the mail in a standard form, and an entirely different
one would fetch the mail in that same standard form.  These standard
forms of delivery were generally integrated into the mail system, which
of course has traditionally been sendmail plus a basic mail program.

Before examining Cyrus, I was looking at using a Maildir format mail
store, letting Postfix deposit mail, and using Courier IMAP to fetch it.
It was suggested to me by a couple people to take a look at Cyrus for
reasons that it had better ACL facilities, and performed better.

What I found out was that Cyrus is rather monolithic in the sense that
it is the same package which deposits mail and fetches it.  It is, as
someone here pointed out, a "black box" with input (LMTP) and output
(IMAP generally).

So I could choose between:

1.  A monolith of MTA + MDA and a separate MAA (Mail Access Agent).
    Postfix and Courier IMAP is what I had considered.

2.  A separate MTA and a monolith of MDA + MAA.  This would be Postfix
    or something else, plus Cyrus.

But what I see is that having a monolith is actually a good thing, not a
bad thing, within reason.  The question, and probably the difference of
opinion, will be just how far that should go.  In my view, Cyrus has
already taken one step towards the monolithic mail server by combining
the Mail Delivery Agent and the Mail Access Agent together.  That appears
to me to have been done for a good reason, which is to enable a custom
mail store architecture to enhance performance over things like a plain
Maildir mail store (which lacks a store of metadata).

It might have been possible to do that without creating a monolith.  But
that would have meant creating yet another standard mail store format and
all the associated documentation of it.  But by being a "black box", it
gets the advantange of being able to change the design of that mail store
at any time without the pressure of standards conformance.  Instead of
standardizing on a format, or even an API, the standard went to a protocol
(LMTP).

What my idea does is carries that concept just one step further.

There is an advantage to having the MDA and MAA integrated.  I believe
there is also an advantage to having the MRA (Mail Receiving Agent, a
subset of MTA functionality) integrated with the MDA.  I don't intent
to carry that to being a full MTA integration ... just an MRA thing.

The problem with this is that the advantages to MRA + MDA are not going
to be realized everywhere.  In some environments (such as EDU), the
issues are simply not the same as in some others.  And even in provider
environments, not everyone is looking at all the cost factors, yet.
There is still a lot of "unlimited mail for one fixed price" attitude
while the people who take advantage of it and email several gigabytes
per day get accounts cut off and peple who mail maybe one or two a week
end up paying the same average price.  While the market isn't crying
out for it, there is a business advantage to being able to provide a
service to both ends of the usage spectrum ... as long as they don't
have to pay for spam.  And there is the issue of being competetive in
the market by making sure your services are more efficient by making
sure your servers don't have to deal with so much mail your customers
don't want to pay for.  These are things the EDU mail administrators
never have to worry about.

Completely monolithic mail server products already exist for Windows.
I know people who are using them.  They do the whole job from SMTP to
IMAP.  I don't know what their implementations actually look like, so
they may be doing things like queueing internally.  I suspect as much
since I do usually get an MDN instead of a DSN for things like an abuse
mailbox not existing.  So their monolithic aspect may be more marketing
than it is implementation.

Back when this whole project was conceived, had I just jumped in and
started to code from scratch, I'm sure that in the 2 years since I
would by now have a working SMTP and IMAP monolithic server.  And I
probably would have a custom mail store to make meta data lookups go
a lot faster.  And I probably would be using Berkeley DB for that and
separate files for content.  It might well be very similar to what
Cyrus is today (although I would have done "virtual" domains from the
beginning ... another concept business needs that the EDU environment
typically does not unless they also do departmental stuff).

My main reason for not wanting to queue mail is to avoid having to do
any Message Disposition Notification.  The reasons for that include to
make sure my server doesn't end sending mail that might be considered
to be spam or similar abuse by someone whose email address was forged
as the sender address by a spammer.  Further, my server could end up
being blacklisted by places like AOL should some spammer try to send
the same spam to thousands of my users; the mass MDN returns could be
treated as a spam burst by AOL and others, triggering an automatic
blacklist.  It's a shame that things have gotten to be that way, but
that is a sad reality now.

In many other cases, the MDN can't be delivered due to a non-responding
MX host, and hence occupies my queue for a while.  That's a cost that
gets passed to my pricing, so I'd rather avoid it.

Almost all legitimate mail comes from an MTA that queues the mail it
is trying to send.  So I see no reason I need to queue it.  If my mail
server is down, I really don't need a secondary MX host to handle it.
Most spamware delivered spam might get lost due to the outage.  That
would be a plus.  So by shedding any need to queue mail, I gain back
more control over the mail services, and can keep costs under control
as well.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam at ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------