Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

Thu Feb 28 18:33:40 EST 2008

Okay, I read over this and I felt worth commenting...

There's mention of using MD, DRBD, LVM2, etc... it sounds extremely  
conviluted and way to complex for what you are needing.

When you are doing a read or a write, each thing takes it's time  
before it gets commited to disk.

If you are doing DRBD, you may want to change a few settings
you're doing raid5 with 3 sata disks using md and drbd... on top of  
lvm etc.

For Example,

<quote>
protocol prot-id
On the TCP/IP link the specified protocol is used. Valid protocol  
specifiers are A, B, and C.

Protocol A: write IO is reported as completed, if it has reached local  
disk and local TCP send buffer.

Protocol B: write IO is reported as completed, if it has reached local  
disk and remote buffer cache.

Protocol C: write IO is reported as completed, if it has reached both  
local and remote disk.
</quote>

 From personal experience, I have found that people usually use  
Protocol C... it's great however it can result in slower writes...  
which depending on your hardware can be very painful.  The fact that  
you have LVM2 sitting in there, as well as MD that means your average  
write has to go through DRBD (and wrote to both servers) ... as well  
as LVM2, and MD before it's actually written... (in a very vague sense)

Additionally you can use LVM2 "Striping" I really won't get into that  
but that may be more beneficial then a RAID-5 with 3 Disks.

There's lots of hints if you read over the archives for speed, I can  
just tell you from what I have read there is nothing you can do with  
your complex setup to make it better.

My best hint for you would be hardware raid, for one that's a big  
step, if you really want raid-5, it may be more beneficial to use 4  
SATA disks... You can and will expect no matter how good your hardware  
is (read slow writes) with RAID-5 and MD.  I had a Zimbra mailserver  
with RAID-5 and the best write I could get was 75Mbit, and that was  
using 8 15k RPM SCSI disks... :(

Hardware Raid, remove LVM unless you really need it... remove DRBD  
unless you totally need it.... there is other ways to create  
redundancy that are better then DRBD... It's not that I hate DRBD... I  
just hate seeing it implemented in places where it just does not  
belong....

I don't know if this will make sense, if it doesn't let me know and  
I'll break it down further if you need it.

Lastly, if you could show us some of your syslog to see if there is  
actually any warnings about '440 lockers in use' or such?

Scott

On Feb 28, 2008, at 2:54 PM, Michael Bacon wrote:

> Jeff,
>
> Just as a rule of thumb, if you've got problems with Cyrus (or any  
> mail
> system), 90% of the time they're related to I/O performance.
>
> I've never seen drbd used for Cyrus, but it looks like other folks  
> have
> done it.  The combination of drbd+lvm2+ext3 might put you somewhere
> unpleasant, but I'll have to let the Linux-heads jump in on that one.
>
> Beyond that, I don't see anything obviously wrong, but maybe someone  
> who's
> run it more on Linux can chime in.
>
> -Michael
>
> --On Thursday, February 28, 2008 3:36 PM -0700 Jeff Fookson
> <jfookson at as.arizona.edu> wrote:
>
>> Michael Bacon wrote:
>>
>>> What database format are you using for the mailboxes database?  What
>>> kind of storage is the "metapartition" (usually /var/imap) on?  What
>>> kind of storage are your mail partitions on?
>>
>> Databases are all skiplist. Our mail partition and the  
>> metapartition are
>> both on the same filesystem, as we intended that both be part of  
>> the same
>> drbd mirror. That partition is
>> a linux software RAID 5 (3 SATA disks). On top of the md layer is the
>> drbd device; on top of that is an lvm2 logical volume; on top of  
>> that is
>> an ext3 filesystem, mounted
>> as '/var/imap'. The mail is then in /var/imap/mail and the metadata  
>> in
>> /var/imap/config (and we also have /var/imap/certs for the ssl  
>> stuff, and
>> /var/imap/sieve for sieve scripts).
>>
>> Thanks.
>>
>> Jeff Fookson
>>
>>>
>>>
>>> --On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
>>> <jfookson at as.arizona.edu> wrote:
>>>
>>>> Folks-
>>>>
>>>> I am hoping to get some help and guidance as to why our  
>>>> installation of
>>>> cyrus-imapd 2.3.9
>>>> is unusably slow. Here are the specifics:
>>>>
>>>> The software is running on a 1.6GHz Opteron with 2Gb memory  
>>>> supporting a
>>>> user base of about 400
>>>> users. The average rate of arriving mail is on the order of 1-2
>>>> messages/sec. The active mailstore
>>>> is about 200GB.  There are typically about 200  'imapd'
>>>> processes at a given time and a hugely varying number of  
>>>> 'lmtpds' (from
>>>> about 6 to many hundreds during
>>>> times of greatest pathology). System load is correspondingly in  
>>>> the 2-15
>>>> range, but can spike to 50-70!
>>>>
>>>> Our users complain that the system is extremely sluggish during  
>>>> the day
>>>> when the system is most busy.
>>>>
>>>> The most obvious thing we observe is that both the lmtpds and the  
>>>> imapds
>>>> are spending HUGE times waiting
>>>> on locks. Even when the system load is only 1-2, an 'strace'  
>>>> attached to
>>>> an instance of lmtpd or imapd shows
>>>> waits of  upwards of 1-2 minutes to get a write lock as shown by  
>>>> the
>>>> example below (this is from a trace of an 'lmtpd')
>>>>
>>>> [strace -f -p 9817 -T]
>>>> 9817  fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
>>>> len=0}) = 0 <84.998159>
>>>>
>>>> We strongly suspect that these large times waiting on locks is  
>>>> what is
>>>> causing the slowness our users are reporting.
>>>>
>>>> We are under the impression that a single instance of cyrus-imapd  
>>>> scales
>>>> well up to about 1000 users (with about 1MB active
>>>> memory per 'imapd' process),  and so we are baffled as to what  
>>>> might be
>>>> going on.
>>>>
>>>> A non-standard aspect of our installation which may have  
>>>> something to do
>>>> with the problem is that we are
>>>> running cyrus on an lvm2 partition that itself is running on top of
>>>> drbd. Thinking that the remote writes
>>>> to the drbd secondary might be causing delays, we put the primary  
>>>> in
>>>> stand-alone mode so that the drbd layer
>>>> was not doing any network activity (the drbd link is running at  
>>>> gigabit
>>>> speed on its own crossover cable to
>>>> the secondary box) and saw no significant change in behavior. Any  
>>>> issues
>>>> due to locking and the lvm2 layer
>>>> would, of course, still be present even with drbd's activity  
>>>> reduced to
>>>> just local writes.
>>>>
>>>> Can anyone suggest what we might do next to debug the problem  
>>>> further?
>>>> Needless to say, our users get
>>>> extremely unhappy when trivial operations in their mail clients  
>>>> take
>>>> over a minute to complete.
>>>>
>>>> Thank you for any thoughts or advice.
>>>>
>>>> Jeff Fookson
>>>>
>>>> --
>>>> Jeffrey E. Fookson, PhD            Phone: (520) 621 3091
>>>> Support Systems Analyst, Principal    jfookson at as.arizona.edu
>>>> Steward Observatory
>>>> University of Arizona
>>>>
>>>> ----
>>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Jeffrey E. Fookson, PhD			Phone: (520) 621 3091
>> Support Systems Analyst, Principal	jfookson at as.arizona.edu
>> Steward Observatory
>> University of Arizona
>>
>
>
>
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
>
> !DSPAM:47c73fda39799180613726!
>
>