improving concurrency/performance

[Gabor Gombas]

On Mon, Nov 07, 2005 at 11:59:39AM +0100, Paul Dekkers wrote:

> >Make sure that you format ext3 partitions with dir_index which improves 
> >large directory performance. 
> 
> ... but decreases read performance in general... at least that is what I 
> found under RH / Fedora!

Yes, processing directory entries in the order returned by readdir() is
slower when dir_index is enabled:

https://listman.redhat.com/archives/ext3-users/2004-September/msg00029.html

Actually you'd get similar slowdown if you create/delete a lot of files
in random order. The speed of readdir() w/ dir_index on a newly
populated directory should be similar to the speed of readdir() w/o
dir_index on a heavily used mail folder.

It would be interesting if you could repeat the measurement with
LD_PRELOAD'ing the readdir-sorting library posted in:

https://listman.redhat.com/archives/ext3-users/2004-September/msg00025.html

> BTW: I found that chaning dir_index with tune2fs didn't work as 
> expected. If I disabled the dir_indexes, even after a forced fsck, 
> performance was still slow. Enabling didn't give predictable results 
> either: I had to specify it with mkfs.

Disabling dir_index will not reorder existing directories. Again, the
result should be similar to what you'd get after creating/deleting a lot
of files in random order, so that the readdir() order no longer matches
the disk layout.

Also, this read benchmark only models the case when users download every
mail in a folder sequentially (like POP3). It does not tell anything
about the case when users randomly open files (by explicit filename
instead of using readdir()) inside a folder. I think Cyrus uses its own
database instead of doing a readdir() even for POP3, so this benchmark
may not even match POP3 usage.

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------