Cyrus infrastructure performance less than expected

[Alain Spineux]

On Mon, Apr 28, 2008 at 11:31 PM, Eric Déchaux <Eric.Dechaux at sun.com> wrote:
> Dear all,
>
>
>  I have juste setup a Cyrus infrastructure based on VMware Virtual
>  Infrastructure 3.0.1 and I have a huge performance problem.
>
>  The new infrastructure was sized for 120 000 mailboxes with 42 000
>  maximum concurrent IMAPS sessions (neither POP3 nor IMAP available). The
>  important point is the concurrent sessions count rise from 6 000 to 42
>  000 in less than an hour, at the beginning of the day when everybody
>  starts to work.
>
>
>  The hardware is made of two Sun Microsystems X4600 chassis with 8 dual
>  core AMD Opteron 885, 64 Gb RAM, 2 dual port 4G QLogic HBA and 4 Intel
>  e1000 NIC (2 being used for ESX System Console and VMotion). LAN backend
>  is 100% Giga Ethernet, SAN backend is 100% 4 Gb FC.
>
>
>  The Virtual machines are running Debian Etch 4.0r3 AMD64 and Cyrus 2.2
>  and are configured as follow :
>
>  On each physical server :
>
>     - 4 frontends : 2 vCPU, 8 Gb RAM, 8 Gb SWAP / Total 8 vCPU, 32 Gb RAM
>     - 3 backends : 4 vCPU, 8 Gb RAM, 8 Gb SWAP, ~500 Gb reizerfs storage
>  through a 550 Gb VMFS LUN / Total 12 vCPU, 24 Gb RAM, 1.5 Tb storage
>     - 1 webmail : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>     - 1 smtp : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>     - 1 mupdate : 2 vCPU, 2 Gb RAM, 2 Gb SWAP
>
>  Total : 26 vCPU, 62 Gb RAM.
>
>
>  This infrastructure is the consolidation of an existing one that runs
>  Debian Sarge 3.0 i386 with 8 frontends and 5 backends. Each server is a
>  dual Intel Xeon 3.0 Ghz with 4 Gb RAM and 4Gb SWAP.
>
>
>  The older infrastructure can stand the 42 000 concurrent sessions, the
>  new one can't : I was expecting each frontend to be able to handle 5 500
>  concurrent sessions but they are not. Around 3 000 / 3 500 concurrent
>  sessions the frontends begin to SWAP and are not more able keep up the load.

At least you know your problem :-) SWAP should be easier to solve than
IO or CPU problem.
I suppose your old frontends where never (or rarely) swapping ! You
have replaced 2 old frontents
with 4Gb by 1 bigger with 8Gb ! Something in the new configuration
make the new frontent
swap!
Do you use 64bits software, 64bits OS ?
Are you sure all the 8Gb are available to your applications or are in use ?

Can you give a list of all configuration and software changes between
old and new configuration ?

Regards.

>
>
>  I don't have any system log that complains about something and the only
>  Cyrus message I got is a DB4 warning about lockers. I think the DB4 in
>  question is the TLS sessions cache DB and in my case the number of
>  lockers can be as high as 8 000...
>
>
>
>  Does somebody have any clue of what I could tune / tweak / configure
>  either on Linux or Cyrus side to make the solution work better ?
>  Is my initial sizing wrong and only solution is to add hardware ?
>
>
>  Many thanks.
>  --
>  Eric
>  ----
>  Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>  Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>  List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you