Load spikes when new email arrives

[Andrew Morgan]

On Wed, 23 Jan 2013, francis picabia wrote:

> Here are more stats.  Do these look average for performance?
> It is difficult to understand why the system was working with few
> load spikes before.
>
> A mailman mailing list sends 10kbyte message to 4000
> users having accounts on this cyrus system.  If I
> grep "Delivered" in the maillog by the minute I can
> see how fast the messages are stored.
>
> e.g.:
> # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l
>    696
>
> That is the best.  This peak event pushed the load to 14
> for 12 minutes, where it averages 604 messages
> delivered to cyrus mailboxes per minute.  Is that
> reasonable for  maximum delivery rate?
>
> I've also backed out the change (yesterday) to
> /sys/block/sda/queue/nr_requests
> I think it was pushing the load higher and there is no advantage
> in my hardware (SAS with Perc 5/i Raid 5 over 4 disk)
> to run with a low value for nr_requests.

You can certainly achieve higher delivery rates, but that all depends on 
your underlying hardware and how you have partitioned your system.

Why don't you start running "iostat -x 5" on the system?  Leave this 
running to give you an idea of the baseline behavior and then look at it 
during periods of high load.  I suspect you will see that your svctm and 
%util will go up dramatically when a large number of messages are being 
delivered.  But, let's not make decisions based on assumptions!  :)

On my Cyrus Murder frontends (3 of them), I have limited LMTP connections 
to 25 in cyrus.conf:

   lmtp          cmd="/usr/local/cyrus/bin/lmtpproxyd" listen="lmtp" proto="tcp4" prefork=0 maxchild=25

This prevents our mail relays (Postfix) from opening too many simultaneous 
LMTP connections, which can cause too much I/O contention.  Take a look 
during your periods of high load to see how many lmtpd processes are 
running.  You may want to limit the number.

 	Andy