Cyrus infrastructure performance less than expected

[Eric Déchaux]

Dear all,


I have juste setup a Cyrus infrastructure based on VMware Virtual 
Infrastructure 3.0.1 and I have a huge performance problem.

The new infrastructure was sized for 120 000 mailboxes with 42 000 
maximum concurrent IMAPS sessions (neither POP3 nor IMAP available). The 
important point is the concurrent sessions count rise from 6 000 to 42 
000 in less than an hour, at the beginning of the day when everybody 
starts to work.


The hardware is made of two Sun Microsystems X4600 chassis with 8 dual 
core AMD Opteron 885, 64 Gb RAM, 2 dual port 4G QLogic HBA and 4 Intel 
e1000 NIC (2 being used for ESX System Console and VMotion). LAN backend 
is 100% Giga Ethernet, SAN backend is 100% 4 Gb FC.


The Virtual machines are running Debian Etch 4.0r3 AMD64 and Cyrus 2.2 
and are configured as follow :

On each physical server :

    - 4 frontends : 2 vCPU, 8 Gb RAM, 8 Gb SWAP / Total 8 vCPU, 32 Gb RAM
    - 3 backends : 4 vCPU, 8 Gb RAM, 8 Gb SWAP, ~500 Gb reizerfs storage 
through a 550 Gb VMFS LUN / Total 12 vCPU, 24 Gb RAM, 1.5 Tb storage
    - 1 webmail : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
    - 1 smtp : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
    - 1 mupdate : 2 vCPU, 2 Gb RAM, 2 Gb SWAP

Total : 26 vCPU, 62 Gb RAM.


This infrastructure is the consolidation of an existing one that runs 
Debian Sarge 3.0 i386 with 8 frontends and 5 backends. Each server is a 
dual Intel Xeon 3.0 Ghz with 4 Gb RAM and 4Gb SWAP.


The older infrastructure can stand the 42 000 concurrent sessions, the 
new one can't : I was expecting each frontend to be able to handle 5 500 
concurrent sessions but they are not. Around 3 000 / 3 500 concurrent 
sessions the frontends begin to SWAP and are not more able keep up the load.


I don't have any system log that complains about something and the only 
Cyrus message I got is a DB4 warning about lockers. I think the DB4 in 
question is the TLS sessions cache DB and in my case the number of 
lockers can be as high as 8 000...



Does somebody have any clue of what I could tune / tweak / configure 
either on Linux or Cyrus side to make the solution work better ?
Is my initial sizing wrong and only solution is to add hardware ?


Many thanks.
--
Eric