Cyrus infrastructure performance less than expected

[Eric Déchaux]

Alain Spineux a écrit :
> On Mon, Apr 28, 2008 at 11:31 PM, Eric Déchaux <Eric.Dechaux at sun.com> wrote:
>   
>> Dear all,
>>
>>
>>  I have juste setup a Cyrus infrastructure based on VMware Virtual
>>  Infrastructure 3.0.1 and I have a huge performance problem.
>>
>>  The new infrastructure was sized for 120 000 mailboxes with 42 000
>>  maximum concurrent IMAPS sessions (neither POP3 nor IMAP available). The
>>  important point is the concurrent sessions count rise from 6 000 to 42
>>  000 in less than an hour, at the beginning of the day when everybody
>>  starts to work.
>>
>>
>>  The hardware is made of two Sun Microsystems X4600 chassis with 8 dual
>>  core AMD Opteron 885, 64 Gb RAM, 2 dual port 4G QLogic HBA and 4 Intel
>>  e1000 NIC (2 being used for ESX System Console and VMotion). LAN backend
>>  is 100% Giga Ethernet, SAN backend is 100% 4 Gb FC.
>>
>>
>>  The Virtual machines are running Debian Etch 4.0r3 AMD64 and Cyrus 2.2
>>  and are configured as follow :
>>
>>  On each physical server :
>>
>>     - 4 frontends : 2 vCPU, 8 Gb RAM, 8 Gb SWAP / Total 8 vCPU, 32 Gb RAM
>>     - 3 backends : 4 vCPU, 8 Gb RAM, 8 Gb SWAP, ~500 Gb reizerfs storage
>>  through a 550 Gb VMFS LUN / Total 12 vCPU, 24 Gb RAM, 1.5 Tb storage
>>     - 1 webmail : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>>     - 1 smtp : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>>     - 1 mupdate : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>>
>>  Total : 26 vCPU, 62 Gb RAM.
>>
>>
>>  This infrastructure is the consolidation of an existing one that runs
>>  Debian Sarge 3.0 i386 with 8 frontends and 5 backends. Each server is a
>>  dual Intel Xeon 3.0 Ghz with 4 Gb RAM and 4Gb SWAP.
>>
>>
>>  The older infrastructure can stand the 42 000 concurrent sessions, the
>>  new one can't : I was expecting each frontend to be able to handle 5 500
>>  concurrent sessions but they are not. Around 3 000 / 3 500 concurrent
>>  sessions the frontends begin to SWAP and are not more able keep up the load.
>>     
>
> At least you know your problem :-) SWAP should be easier to solve than
> IO or CPU problem.
> I suppose your old frontends where never (or rarely) swapping ! You have replaced 2 old frontents with 4Gb by 1 bigger with 8Gb ! Something in the new configuration
> make the new frontent swap!
>   
On current production platform the frontends use swap massively but the 
impact is far less than on the new platform.
> Do you use 64bits software, 64bits OS ?
>   
We are indeed using 64bits OS and 64bits software. This is the main 
difference between the old platform which runs a 32 bits software on a 
32 bits OS.
> Are you sure all the 8Gb are available to your applications or are in use ?
>   
I think they are but maybe you can give me a clue to check this and be 
sure ?
> Can you give a list of all configuration and software changes between
> old and new configuration ?
>   
Except moving from i386 to AMD64, the software changes are the one 
included in Debian Etch. This include moving from a 2.6.9 kernel (I 
think, not sure though) to a 2.9.18.

Cyrus version went from 2.1 to 2.2.
> Regards

Many thanks.
--
Eric