Graceful degradation in overload conditions

[Eric Luyten]

On Sat, April 4, 2009 3:19 pm, Gary Mills wrote:
> We had a situation recently where our Cyrus back-end server
> encountered a memory shortage.  The initial result was slow response, but
> that provoked an increase in the number of imapd, pop3d, and lmtpd
> processes, making the problem much worse.  The lmtpd processes accumulated
> because of slow delivery of incoming e-mail, but the others were likely a
> result of users starting more sessions.  The results were not pretty!
>
> What can be done to provide a graceful degradation of service in
> overload conditions, so that the server protects itself against a resource
> shortage?  I did put an upper limit on lmtpd children. Sendmail will just
> queue incoming messages when this limit is reached.

Gary,

For what it's worth : our server, a Solaris 9 V1280 with 32 Gigs of RAM
and eight processors, serves 55k users owning 400k mailboxes, 32M msgs
making for 2.7 TB of mailspool content.
Last week we registered the following number of Postfix deliveries into
the Cyrus spool :

   Mon 207878
   Tue 190345
   Wed 183913
   Thu 175530
   Fri 149533
   Sat  55441
   Sun  49292

I have the lmtp process limit set to 10 (ten), coming down from 20, which
make for a bit too many "DBERROR db3: NN lockers" warnings.
Our delivery DB is a bottleneck ; I'm planning carving up our Cyrus into
multiple instances, then moving to multiple (front/back) servers, this
summer, also upgrading from 2.2.13 to 2.3.LATEST

Cheers,
Eric Luyten, Computing Centre, Brussels Free Universities.