I forwarded John's message to Andrew Morton, linux kernel maintainer, and
this is his reply (it was cc'ed to the list, but, not being a subscriber,
I presume it bounced):
------- Forwarded Message
Date: Tue, 08 Nov 2005 15:21:31 -0800
From: Andrew Morton <akpm at osdl.org>
To: Andrew McNamara <andrewm at object-craft.com.au>
cc: "John Madden" <jmadden at ivytech.edu>,
info-cyrus at lists.andrew.cmu.edu
Subject: Re: improving concurrency/performance (fwd)
Andrew McNamara <andrewm at object-craft.com.au> wrote:
>
> This guy is having a problem with cyrus-imap and ext3 - when multiple
> processes are attempting to write to the one filesystem (but not the one
> file), performance drops to next to nothing when only five processes are
> writing. An strace shows most of the time is being spent in fdatasync
> and fsync.
>
> ...
Yes, on ext3, an fsync() syncs the entire filesystem. It has to, because
all the metadata for each file is shared - it's just a string of
journallable blocks. Similar story with the data, in ordered mode.
So effectively, fsync()ing five files one time each is performing 25 fsync()s.
One fix (which makes the application specific to ext3 in ordered-data or
journalled-data mode) is to perform a single fsync(), with the
understanding that this has the side-effect of fsyncing all the other
files. That's an ugly solution and is rather hard to do if the workload
consists of five separate processes!
So I'd recommending mounting the filesystem with the `-o data=writeback'
mode. This way, each fsync(fd) will sync fd's data only. That's much
better than the default data-ordered mode, wherein a single fsync() will
sync all the other file's data too.
In data=writeback mode it is still the case that fsync(fd) will sync the
other file's metadata, but that's a single linear write to the journal and
the additional cost should be low.
Bottom line: please try data=writeback, let me know.
> ------- Forwarded Message
>
> Date: Tue, 08 Nov 2005 09:25:54 -0500
> From: "John Madden" <jmadden at ivytech.edu>
> To: Jure =?iso-8859-1?Q?Pe=E8ar?= <pegasus at nerv.eu.org>
> cc: info-cyrus at lists.andrew.cmu.edu
> Subject: Re: improving concurrency/performance
>
> > As expected, these are from locking operations. 0x8 is file descriptor,
> > which, if I read lsof output correctly, points to config/socket/imap-0.lock
> > (what would that be?) and 0x7 is F_SETLKW which reads as "set lock or wait
> > for it to be released" in the manual page.
>
> Yup, that's exactly the sort of thing I was suspecting -- the performance I
> was seeing just didn't make sense.
>
> imap-0.lock is in /var/imap/socket for me. I believe it's one of the lock
> file s created when cyrus is started, so it wouldn't make any sense for
> imapd to ever be spinning on it.
>
> The delays I was seeing ocurred when multiple imapd's were writing to the
> spool at the same time. I do see a lot of this though:
>
> fcntl(6, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
>
> It looks like the lock to open a file in the target mailbox. But again,
> very l ow actual throughput and still little or no iowait. However, adding
> a -c to the strace, the top three syscalls are:
>
> % time seconds usecs/call calls errors syscall
> - ------ ----------- ----------- --------- --------- ----------------
> 52.68 0.514720 1243 414 fdatasync
> 29.87 0.291830 846 345 fsync
> 4.19 0.040898 27 1519 fcntl
>
> Makes me wonder why the fsync's are taking so long since the disk is
> performing so well. Anyone know if that's actually typical?
>
> --
> John Madden
> UNIX Systems Engineer
> Ivy Tech Community College of Indiana
> jmadden at ivytech.edu
>
> - ----
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
> ------- End of Forwarded Message
------- End of Forwarded Message