Cyrus process model...

[Rob Mueller]

[ Continued from an off mailing list conversation about killing cyrus lmtpd
processes when they go haywire, and cyrus process accounting ]

> > Surely this is a relatively well solved problem? Just about every unix
> > system uses this master/forked child approach? How does apache do it?
> > Net::Server::PreFork? I can't imagine that there aren't cookbook
solutions
> > to this issue since it's what unix has been doing for 30 years? Or is
there
> > something I'm missing here?

> There are many different possibilities. Most other systems limit the
> number of clients instead of forking new processes on demand without a
> set limit. Apache also doesn't have differentiated children or
> substantial shared state. (All children are members of the same
> service or you don't particularly care how many additional unused children
> you have...)

I was under the impression that Apache 2 was planning on making it's
forking/threading model much more generic, and supporting a general
'services' model, including a library to abstract the underlying OS? Hmmm,
looking into that, it appears that it's mostly done already.

http://apr.apache.org/
http://apr.apache.org/apr2_0intro/apr2_0intro.htm

And more:

Contains following functionality
-Reading and writing of files
-Character set conversion
-Network communications using sockets
-Time management used for Internet type conversions
-String management like C++ including natural order management
-UNIX Password management routines
-Table management routines
-UUID Internet generation
-Filename canonicalization
-Random data generation
-Global lock management
-Threads and process management
-Dynamic library loading routines
-Memory mapped and shared memory

-----

http://www.arctic.org/~dean/apache/2.0/process-model.html

I think the above is general enough to implement the interesting process
models, and to implement optimizations that are available only in some of
the multi-threaded models. Note that nothing above is specific to HTTP, and
I believe that we should strive to keep the abstraction so that the same
libraries can be used to implement other types of servers (i.e. FTP,
streaming video/audio, corba).

-----

Would the cyrus team think it worthwhile to consider refactoring to use the
new Apache 2 APR modules? I know off hand that it would be a lot of work,
but it could be a gradual re-factoring process, and the idea of actually
reusing code between projects would be *really* nice.

Joel Spolsky is a big proponent of refactoring over time to improve software
and you can read some of his thoughts here.

http://www.joelonsoftware.com/articles/fog0000000069.html
http://www.joelonsoftware.com/news/fog0000000328.html

Ooops, I'm feeling a rant come along...

*** RANT MODE ***

I know this is a little off topic, but the source for cyrus is really
showing it's age a bit. I know that happens with all software, you start
with certain assumptions, and the more you go on, the more the original
assumptions get blown away, so you hack this in here, and there, and then
every now and then, you go on a big cleanup spree! The problem I feel is
that the cleanup hasn't been big enough or often enough.

Also, over time programming habits change. Many old C idioms are pretty much
dead. Most of the C string handling methods are now annoying, or downright
dangerous. There are several dozen replacement libraries, including the APR
one above, and good ones like
http://www.annexia.org/freeware/c2lib/index.msp. This library also
implements automatically resizing arrays and memory pools, a common way to
avoid all subtle leaks introduced by malloc() and the like, and to avoid the
buffer overflows of stack buffers.

I'm sure I could go on and on, and I'd eventually get back to the fact that
the biggest problem is that the original C lanuage and library is pretty
horrible in hindsight. But people in general still only dare to use the
original library and idioms, and loath to include extra dependencies in
their products, often instead rewriting the same new set of libraries from
scratch! *sigh* Ok, better stop now...

*** RANT MODE OFF ***

Sorry, don't mean for any of this to come across as too negative. Actually,
I'm hoping to inspire people enough to look into some of these alternatives,
and see if there can at least be some pushes into what I feel would be a
better direction for the coding of cyrus to go. Of course, it's only IMHO...

Rob