Just in case it is of general interest: ZFS mirroring was the culprit in our case

[Pascal Gienger]

Our latency problems went away like a miracle when we detached one half of 
the mirror (so it is no more a mirror).

Read-Rates are doubled (not per device, the total read rate!), latency is 
cut off. No more latency problems.

When attaching the volume again, resilvering puts the system to a halt - 
reads and writes do block for seconds (!). We will go on directly with Sun 
to solve the problem. Their "lowest I/O-priority to resilver disks" does 
not seem to be effective. It really blocks the kernel and you end up with 
thousand locks in "zfs_zget".

We have two SAN volumes in different buildings which are NOT the 
bottleneck, tests show it.

Pascal