Slow lmtpd

[Andre Nathan]

Hello

We have a cyrus server that runs under heavy load, and a few days ago it
started to show a behaviour where its lmtpd processes take a long time
to deliver messages sent from postfix. Below is an example of a postfix
log message. with the email address removed:

Feb 28 17:14:11 mta13 postfix/lmtp[9633]: 63CBA700071C:
to=<user=domain.com at box2.prv.f3.k8.com.br>,
relay=box2.prv.f3.k8.com.br[10.8.5.49]:2003, conn_use=6, delay=1718,
delays=0.15/1118/0/600, dsn=4.4.2, status=deferred (conversation with
box2.prv.f3.k8.com.br[10.8.5.49] timed out while sending end of data --
message may be sent more than once)

The last field in the "delays" field shows that the time out occurred
after 600s trying to send the message to cyrus. Even when a timeout does
not occur, the time for the message to be sent is around 100-300s.

We have some other more lightly loaded servers in which the data is sent
to lmtpd very quickly, and at any given moment there are only a few
lmtpd's running. On the heavily loaded servers, there are always a few
thousand lmtpd processes.

I've run strace on lmtpd processes on both servers, and I couldn't spot
any significative difference in their behaviours. They both run
cyrus-imapd 2.2.12.

So, I'm looking for some help and suggestions in trying to debug this
situation, and maybe some tuning options to for lmtpd, or at least find
out why they're taking so long to do complete the message delivery. The
load on this server has always been pretty high, but this behaviour only
started recently. It has much more mailboxes than the other server, as
it mounts its volumes from a large AoE storage device.

Thanks in advance,
Andre