Just in case it is of general interest: ZFS mirroring was the culprit in our case
Pascal.Gienger at uni-konstanz.de
Tue Nov 13 01:23:06 EST 2007
Our latency problems went away like a miracle when we detached one half of
the mirror (so it is no more a mirror).
Read-Rates are doubled (not per device, the total read rate!), latency is
cut off. No more latency problems.
When attaching the volume again, resilvering puts the system to a halt -
reads and writes do block for seconds (!). We will go on directly with Sun
to solve the problem. Their "lowest I/O-priority to resilver disks" does
not seem to be effective. It really blocks the kernel and you end up with
thousand locks in "zfs_zget".
We have two SAN volumes in different buildings which are NOT the
bottleneck, tests show it.
More information about the Info-cyrus