Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

Just a little note, for those who have perhaps the same problem.

We saw performance problems after we switched from a Linux installation to 
a Solaris 10 cluster connected to our SAN (using scsi_vhci and 2 Qlogic 
Controllers).

Problems arose when real load came to the machine, despite having tested it 
with some load simulation scenarios...

Looking at the attached chart[0], you will notice up to 10 seconds time to 
select a mailbox, far too long.

The problem was ZFS - ZFS is perfect for rubbish, cheap disks with ugly 
firmwares. So if your storage is "too good" it makes things go bad.

First: The file prefetch algorithm does not seem to be very good for 20 
million mail files and 300,000 cyrus meta files... ;-)

So first step was (at 11am in the chart) to disable this prefetching 
routine. [1]

This cut off 4 seconds from requests, but still 10 second-timeouts were 
seen.

Next step was to disable the zfs cache flush. As said, ZFS is good for 
rubbish disks, so every 5 seconds it instructs the SATA or SCSI drives to 
flush their internal ram to disk. This is good for cheap disks, but a no-go 
if you have a SAN RAID storage having 2 Gigabytes and more of RAM storage, 
buffered with a battery. In fact, our storage system really flushed all 5 
second its complete RAM cache, you even saw it watching the "blinkenlights".
Plus, every fsync() call did the same... :(

There is a trick: You can disable this in ZFS [2]. You see the result at 
4pm (16:00 european notation in the chart). CHILDREN DO NOT TRY THIS AT 
HOME..... Only do this if you don't have any "real" physical disk storages 
attached to your system with zfs pools on them - otherwise you will lose 
data on power outages as the ram cache of your hard disk is not buffered by 
a battery.

Now the machine runs and handles all mail without noticeable delays.

[0] http://priscilla.rz.uni-konstanz.de/mailserver/

[1] in /etc/system:   set zfs:zfs_array_rd_sz=0
    on a live system using mdb -kw: zfs_array_rd_sz/Z0x0

[2] in /etc/system:   set zfs:zfs_nocacheflush=1
    on a live system using mdb -kw: zfs_nocacheflush/W0t1

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mailselect.png
Type: image/png
Size: 18292 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20070920/cc71f061/attachment-0001.png