Graceful degradation in overload conditions

[Andrew Morgan]

On Sat, 4 Apr 2009, Gary Mills wrote:

> We had a situation recently where our Cyrus back-end server
> encountered a memory shortage.  The initial result was slow response,
> but that provoked an increase in the number of imapd, pop3d, and lmtpd
> processes, making the problem much worse.  The lmtpd processes
> accumulated because of slow delivery of incoming e-mail, but the
> others were likely a result of users starting more sessions.  The
> results were not pretty!
>
> What can be done to provide a graceful degradation of service in
> overload conditions, so that the server protects itself against a
> resource shortage?  I did put an upper limit on lmtpd children.
> Sendmail will just queue incoming messages when this limit is reached.
>
> What about imapd and pop3d daemons, which also consume resources?
> Are limits a good idea here too?  Users will complain to the help
> desk when those limits are reached, of course.  Can the msg/shutdown
> file be used to control imapd processes in a nicer manner?

You can set the lmtpd limits pretty low without affecting mail delivery. 
We limit to 100 here, which is still probably too high.

The limit on imapd processes really depends on your hardware.  If you can 
make some reasonable guess about the incremental amount of memory used by 
each imapd, then you could set the limit to something that wouldn't 
exhaust all the memory on the server.  That's what I've done here.  We set 
the limit to 1500 imapds processes on our 4GB server.  Our normal load is 
around 700 imapds, so that leaves lots of memory for caching I/O.

 	Andy