Load spikes when new email arrives

[francis picabia]

A pictures says a thousand words...  Here is the pattern we see lately:

[image: Inline image 2][image: Inline image 1]

Green indicates active messages in the postfix mailqueue, inbound.
The load is negligible until new mail arrives.

There are not many users on the system at that hour but there could
be about 400 mostly idle IMAP connections from webmail (another server)
and smartphones, etc.  There are 4800 mailboxes, stored on
a partition dedicated to cyrus.

We use lmtp with postfix.  In cyrus.conf:

lmtpunix    cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1

In main.cf:

mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp

All of the scanning like clamav, amavis, etc.
happens at the MX before it hits this cyrus server.

The system is a single cyrus box, 8 GB RAM, dual Xeon 3.2 Ghz
(4 cores total) and 4 SAS drives with Perc 5 controller providing RAID 5.

free command and a cacti graph of memory use shows we have lots
of free RAM.

This system has run since 2009 without issues.  It is Redhat Enterprise
Linux 5.9
(education version, no support).

The problem appears quickly, but when I've seen it, the iowait time
goes as high as 50%, but briefly.

I've checked some articles about tuning for cyrus and switched to deadline
scheduler and lower nr_requests value of 4.  It may have helped a little but
nothing very significant.

We run these from cyrus.conf:

EVENTS {
  squatter cmd="squatter -s -r user" at=0410
  checkpoint    cmd="ctl_cyrusdb -c" period=30
  delprune    cmd="cyr_expire -E 3" at=0400
  tlsprune    cmd="tls_prune" at=0400
}

We're using the postfix/cyrus packages directly from Redhat.
postfix 2.3.3
cyrus-imapd 2.3.7

I'm wondering if there is anything more I can check/tweak.

Is there maintenance we should consider like rebuilding
all mailboxes?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20130121/a092a48e/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 10759 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20130121/a092a48e/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 11198 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20130121/a092a48e/attachment-0003.png