fsync() takes about 0.06 second ---makes Cyrus deliver slow

[Su Li]

Hi,

I am trying the speed up the speed of deliver mail. After I took David's
suggestion, did chattr -S on /spool/imap. writev() function takes 0.008
seconds. But the fsync() function call still takes 0.06 second and for each
new email there are 3-4 fsync(). So for each new email, Cyrus needs 0.8
seconds to save to local mail store. If there is a big number of emails
coming in, there will be thousands of processes created and the DBERROR: 399
locks ... will show up. I know the imap server is too slow to keep up with
sendmail.  

Larry, 
I saw a lot of emails from talking about fsync(), would you comment on this?


Here is the configuration of my Cyrus IMAP:

-Linux 7.2, dual CPU with 2G memory, RAID 5 hard disks

-BerkeleyDB.4.0, cyrus-imapd-2.1.8, cyrus-sasl-2.1.7

-imap configure file: configure --with-auth=unix --without-krb4
--without-ucdsnmp --with-bdb-incdir=/usr/local/BerkeleyDB.4.0/include
--with-bdb-libdir=/usr/local/BerkeleyDB.4.0/lib --with-dblib=berkeley
--with-dbpath=/usr/local/BerkeleyDB.4.0 --with-lock=fcntl
--with-mboxlist-db=skiplist --with-seen-db=skiplist
--with-duplicate-db=db3_nosync --without-notify --disable-sieve
--enable-fulldirhash --with-idle=no

Here is the result for strace -T -p lmtppid:

open("/var/spool/imap/user/loadtest711/cyrus.index", O_RDWR) = 12 <0.000015>
fstat64(12, {st_mode=S_IFREG|0600, st_size=5120, ...}) = 0 <0.000006>
mmap2(NULL, 16384, PROT_READ, MAP_SHARED, 12, 0) = 0x41749000 <0.000012>
open("/var/spool/imap/user/loadtest711/cyrus.cache", O_RDWR) = 13 <0.000015>
fstat64(13, {st_mode=S_IFREG|0600, st_size=61264, ...}) = 0 <0.000007>
mmap2(NULL, 73728, PROT_READ, MAP_SHARED, 13, 0) = 0x4174d000 <0.000012>
.......

writev(13, [{"\0\0\0014(\"Mon, 9 Sep 2002 12:35:44 -"..., 312},
{"\0\0\0Q(\"TEXT\" \"PLAIN\" (\"CHARSET\" {"..., 88}, {"\0\0\0E(\"TEXT\"
\"PLAIN\" (\"CHARSET\" {"..., 76},
{"\0\0\0000\0\0\0\2\0\0\0\0\0\0\2V\0\0\2V\0\0\0000\377\377"..., 52},
{"\0\0\0\0", 4}, {"\0\0\0)loadtest <loadtest711 at web9.s"..., 48}, {"\0\0\0
<loadtest711 at web9.swlab.rim."..., 36}, {"\0\0\0\0", 4}, {"\0\0\0\0", 4},
{"\0\0\0\3NIL\0", 8}], 10) = 632 <0.008176>
munmap(0x40400000, 646)                 = 0 <0.000011>
fsync(14)                               = 0 <0.044641>
close(14)                               = 0 <0.000008>
lseek(12, 5120, SEEK_SET)               = 5120 <0.000007>
write(12, "\0\0\0b=|\315\340=|\305\200\0\0\2\206\0\0\2V\0\0\2V\0\0"..., 52)
= 52 <0.000018>
fsync(12)                               = 0 <0.065687>
fsync(13)                               = 0 <0.005975>


We can see fsync(12), which is processing cyrus.index took 0.06 second, but
fsync(13), which is processing cyrus.cache only took 0.0059 second. Why
there is such big difference between the file cyrus.index and cyrus.cache?

I wonder is there any way to may fsync() faster? If not is there any way to
call fsync less often? Will using a fast hard disk help? Or will move the
mail store -- /spool/imap/user to a database help?


Su 

Research In Motion
(519)888-7465 ext 3041