rooling replication and many pop3d
Oberdan Albertoni
obe at digicolor.net
Thu Nov 24 05:52:00 EST 2011
On 24/11/2011 02:07, Andy Bennett wrote:
> Hi,
>
>>>> I enabled replication between two servers with version 2.4.10 cyrus.
>>>> I set the option for the rolling replication, and it works fine but
>>>> obviously I have a high CPU load.
>>>> Unfortunately after 10 minutes of running processes pop3d increasing
>>>> from 50 to over 200, making the server unusable for customers.
>>>> Can you tell me why this increase is abnormal?
>>> Can you use something like 'top' to work out which processes are
>>> consuming most of the CPU time?
> Thanks for the screenshots.
>
>
>> This screenshot of top before :
>>
>> http://www.digicolor.net/cyrus/img1.jpg
> This shows a load average of around 1. That means that, at any given
> point in time over the past 1 and 5 minutes, 1 process has been waiting
> in the run queue, ready to go.
> This therefore not an entirely idle machine. I see you're running a
> nameserver and a few other things: it looks they've been busy on the CPU
> but not excessively so.
>
> What's more worrying is the 4.5% of CPU time spent "waiting". This time
> is accrued when processes are unable to run due to outstanding IO.
>
>
>
>> and after 20 minutes of rooling replication :
>>
>> http://www.digicolor.net/cyrus/img5.jpg
> This shows a load average of around 7 and 30% of CPU time spent in
> iowait. This machine does not seem to be managing well with the IO load
> of rolling replication.
>
>
>
>>> Can you use something like 'vmstat 1' to show us how much I/O there is
>>> on the system?
>> This screenshot of top before :
>>
>> http://www.digicolor.net/cyrus/img3.jpg
> This shows a system that is not reading anything from disk (bi). A small
> number of blocks are being written out to disk (bo). Each line
> represents activity for a period of 1 second, as specified by the
> parameter to 'vmstat'.
>
>
>> and after 20 minutes of rooling replication :
>>
>> http://www.digicolor.net/cyrus/img6.jpg
> This system is writing to disk but it's very choppy. Sometimes it's
> getting 7,000 blocks out per second and other times it's only 1,000.
> Depending on your block size, this probably represents only a few MB per
> second. The last column shows iowait CPU percentage.. and it's rather large.
>
> What IO subsystem do you have on this machine? What filesystem are you
> using?
I have 4 HDD 500GB S SATA II
with LSI 8704 ELP SAS
RAID 6
filesystem ext3 mounted with noatime, nodiratime
> The IO on this machine appears to be struggling significantly.
>
> I did a quick test on my laptop. I have a 2.5", 7,200rpm 200GB disk.
>
> I ran this in my home directory to cause every file to be read from disk:
>
> -----
> $ find -type f | xargs cat> /dev/null
> -----
>
> 'vmstat 1' gives lines like this:
>
> -----
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 1 1 1680 30600 272 2575688 0 0 12376 576 2141 3424 3 6
> 47 43
> 1 1 1680 32396 272 2576736 0 0 29988 0 1882 3632 5 7
> 47 41
> 0 1 1680 33868 272 2576660 0 0 46820 0 2304 4443 4 8
> 48 39
> 1 0 1680 33416 272 2578600 0 0 36716 0 2067 3733 3 7
> 48 42
> 0 1 1680 34000 272 2581944 0 0 50432 0 1164 2983 3 6
> 50 42
> 0 1 1680 31876 272 2585320 0 0 46464 64 1223 2964 3 8
> 49 40
> 1 1 1680 30288 272 2588672 0 0 51712 0 1380 3658 3 7
> 46 43
> 0 1 1680 29836 272 2590552 0 0 59776 0 1288 3549 4 7
> 47 42
> 0 1 1680 30324 272 2592948 0 0 58368 0 1287 3568 2 7
> 49 41
> 1 1 1680 30308 272 2593108 0 0 12800 18 917 1673 2 2
> 49 46
> -----
>
> They're an order of magnitude greater than what you're seeing. As you
> can see, I drop a few bi when I start to do bo but that's because I've
> only got a single spindle.
>
> Please can you run the same test?
this my vmstat after I run the command find -type f | xargs cat >
/dev/null :
http://www.digicolor.net/cyrus/img7.jpg
>
> Can you track the source of all those writes in img3?
Please can you tell us more about the type of machine you are trying to
run this on? Thanks for the info and screenshots so far.
Is a :
n. 2 XEON Clov. 2,33 GHz - X5345
8GB RAM
With debian 5.0
Regards,
Obe.
--
Oberdan Albertoni
-----------------------------------------------------------------------------------------------------------------------------------------------------------
DIGICOLOR.NET s.r.l. Via Ala Ponzone, 30 - 26100 CREMONA (CR) Tel. +39 (0372) 30669 - 1965101 - 1951509
Fax +39 (0372) 464840 E-mail info at digicolor.net - Website http://www.digicolor.net
C.F / P.IVA 01227520192 - R.E.A. CR n. 153382 - Capitale Sociale € 10.400 i.v.
Filiale di Codogno
Via Galilei, 17 – 26845 CODOGNO (LO) Tel. +39 (0377) 741006 – 431931 Fax +39 (0372) 464840
E-mail info.codogno at digicolor.net - Website http://www.digicolor.net
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Ai sensi del D.lgs n.196 del 30.06.03 (Codice Privacy) si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo
del destinatario. Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a
terzi, dandocene gentilmente comunicazione. Grazie
This message, for the D.lgs n.196 / 30.06.03 (Privacy Code), may contain confidential and/or privileged information. If you are not the addressee or
authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein.
If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
-----------------------------------------------------------------------------------------------------------------------------------------------------------
More information about the Info-cyrus
mailing list