improving concurrency/performance (fwd)

Andrew McNamara andrewm at
Tue Nov 8 22:48:20 EST 2005

I forwarded John's message to Andrew Morton, linux kernel maintainer, and
this is his reply (it was cc'ed to the list, but, not being a subscriber,
I presume it bounced):

------- Forwarded Message

Date:    Tue, 08 Nov 2005 15:21:31 -0800
From:    Andrew Morton <akpm at>
To:      Andrew McNamara <andrewm at>
cc:      "John Madden" <jmadden at>,
	 info-cyrus at
Subject: Re: improving concurrency/performance (fwd)

Andrew McNamara <andrewm at> wrote:
> This guy is having a problem with cyrus-imap and ext3 - when multiple
> processes are attempting to write to the one filesystem (but not the one
> file), performance drops to next to nothing when only five processes are
> writing. An strace shows most of the time is being spent in fdatasync
> and fsync.
> ...

Yes, on ext3, an fsync() syncs the entire filesystem.  It has to, because
all the metadata for each file is shared - it's just a string of
journallable blocks.  Similar story with the data, in ordered mode.

So effectively, fsync()ing five files one time each is performing 25 fsync()s.

One fix (which makes the application specific to ext3 in ordered-data or
journalled-data mode) is to perform a single fsync(), with the
understanding that this has the side-effect of fsyncing all the other
files.  That's an ugly solution and is rather hard to do if the workload
consists of five separate processes!

So I'd recommending mounting the filesystem with the `-o data=writeback'
mode.  This way, each fsync(fd) will sync fd's data only.  That's much
better than the default data-ordered mode, wherein a single fsync() will
sync all the other file's data too.

In data=writeback mode it is still the case that fsync(fd) will sync the
other file's metadata, but that's a single linear write to the journal and
the additional cost should be low.

Bottom line: please try data=writeback, let me know.

> ------- Forwarded Message
> Date:    Tue, 08 Nov 2005 09:25:54 -0500
> From:    "John Madden" <jmadden at>
> To:      Jure =?iso-8859-1?Q?Pe=E8ar?= <pegasus at>
> cc:      info-cyrus at
> Subject: Re: improving concurrency/performance
> > As expected, these are from locking operations. 0x8 is file descriptor,
> > which, if I read lsof output correctly, points to config/socket/imap-0.lock
> > (what would that be?) and 0x7 is F_SETLKW which reads as "set lock or wait
> > for it to be released" in the manual page.
> Yup, that's exactly the sort of thing I was suspecting -- the performance I
> was seeing just didn't make sense.
> imap-0.lock is in /var/imap/socket for me.  I believe it's one of the lock
> file s created when cyrus is started, so it wouldn't make any sense for
> imapd to ever be spinning on it.
> The delays I was seeing ocurred when multiple imapd's were writing to the
> spool at the same time.  I do see a lot of this though:
> fcntl(6, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
> It looks like the lock to open a file in the target mailbox.  But again,
> very l ow actual throughput and still little or no iowait.  However, adding
> a -c to the strace, the top three syscalls are:
> % time     seconds  usecs/call     calls    errors syscall
> - ------ ----------- ----------- --------- --------- ----------------
>  52.68    0.514720        1243       414           fdatasync
>  29.87    0.291830         846       345           fsync
>   4.19    0.040898          27      1519           fcntl
> Makes me wonder why the fsync's are taking so long since the disk is
> performing so well.  Anyone know if that's actually typical?
> -- 
> John Madden
> UNIX Systems Engineer
> Ivy Tech Community College of Indiana
> jmadden at
> - ----
> Cyrus Home Page:
> Cyrus Wiki/FAQ:
> List Archives/Info:
> ------- End of Forwarded Message

------- End of Forwarded Message

More information about the Info-cyrus mailing list