fsync() takes about 0.06 second ---makes Cyrus deliver slow

[Jeremy Rumpf]

Hmmm, this is an interesting thread. 

Using ext3 or another journaling system might be a good idea if your mail 
store size is large, ie +500G. In the event of a crash, fsck time would be 
tremendous.

One cool thing about ext3 is that you have good control over its journaling 
semantics. I'm assuming your using the default mode of "ordered", which is a 
good all around choice for maintaining metadata and data consistency.

When using ext3 in its default "ordered" mode, metadata is logged, while data 
is synced before the metadata is flushed. This may be a little redundant 
since cyrus is exclusively flushing data with fsync (I'm outta my league a 
bit here).

I would be really curious for you to run your tests with ext3 running in 
"writeback" journaling mode. You can change modes by modifying your fstab and 
remounting the file system:

LABEL=/var   /var   ext3   defaults   1 2

to

LABEL=/var   /var   ext3   defaults,data=writeback   1 2

Some good articles on understanding ext3 are:

http://www.linuxplanet.com/linuxplanet/reports/4136/1/
http://www-106.ibm.com/developerworks/linux/library/l-fs7/

Also, you previously stated that your using a RAID subsystem to store data on. 
Ask your system admin, or perhaps you are the admin :), if the cache on the 
controller is set for "writeback" mode vs "writethrough" mode. Having a 
battery backed cache on the RAID controller + writeback operation could speed 
up the fsync calls since data will be synchronously flushed to the RAID 
controller's cache vs physically being flushed to the disks. This would only 
be possible when running the controller's cache in writeback mode.

Some reading on this is at:

http://www.storageadmin.com/Articles/Index.cfm?ArticleID=5398

Hope this helps,
Jeremy



On Tuesday 10 September 2002 10:31 am, Su Li wrote:
> Daniel,
>
> My Linux is red hat 7.2-1 with kernel 2.4.7-10. The file system is ext3.
> Hard disk it RAID5.
>
> Jure,
> Thanks for your configuration. I 'll give a try and post my result to the
> new group.
>
> David,
> I tried remove suppression from /etc/cyrus.conf. The suppression is still
> there. Maybe I should try remove it from the code.
>
>
> -----
> Besides, I did some calculation on the time of the system calls on a
> new-email. 3 fcync takes about 0.15 seconds. Each mail overall takes 0.65
> seconds. I see the number of locks and deliver process can go up to
> thousands. Maybe some thing else is also taking big time, say the
> communication between deliver and lmtpd?
>
> Anyway, I'll keep trying. Thanks for every body's help. If anybody got some
> idea, please let me know.
>
>
> Su
>
>