<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br><div><div>On 20 Nov 07, at 1756, David Lang wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br></div><div>however a fsync on a journaled filesystem just means the data needs to be </div><div>written to the journal, it doesn't mean that the journal needs to be flushed to </div><div>disk.</div><div><br></div><div>on ext3 if you have data=journaled then your data is in the journal as well and </div><div>all that the system needs to do on a fsync is to write things to the journal (a </div><div>nice sequential write),</div></blockquote><div><br class="webkit-block-placeholder"></div><div>Assuming the journal is on a distinct device and the distinct device can take the load. It isn't on ZFS, although work is in progress. One of the many benefits of the sadly underrated Solaris Disksuite product was the metatrans devices, which at least permitted metadata updates to go to a distinct device. When the UFS logging code went into core Solaris (the ON integration) that facility was dropped, sadly. My Pillar NFS server does data logging to distinct disk groups, but mostly --- like such boxes tend to do --- relies on 12GB of RAM and a battery. A sequential write is only of benefit if the head is in the right place and the platter is at the right rotational position and the write is well-matched to the transfer rate of the spindle: if the spindle is doing large sequential writes while also servicing reads and writes elsewhere, or can't keep up with writing tracks flat out, the problems increase.</div><blockquote type="cite"><div><br></div><div>for cyrus you should have the same sort of requirements that you would have for </div><div>a database server, including the fact that without a battery-backed disk cache </div><div>(or solid state drive) to handle your updates, you end up being throttled by </div><div>your disk rotation rate (you can only do a single fsync write per rotation, and </div><div>that good only if you don't have to seek), RAID 5/6 arrays are even worse, as </div><div>almost all systems will require a read of the entire stripe before writing a </div><div>single block (and it's parity block) back out, and since the stripe is </div><div>frequently larger then the OS readahead, the OS throws much of the data away </div><div>immediatly.</div><div><br></div><div>if we can identify the files that are the bottlenecks it would be very </div><div>interesting to see the result of puttng them on a solid-state drive.</div></blockquote><br></div><div>I've split the meta-data out into separate partitions. The meta data is stored in ZFS filesystems in a pool which is a RAID 0+1 4 disk group with SAS drives, the message data is coming out of the lowest QoS on my Pillar. A ten second fsstat on VM operations shows that by request (this measures filesystem activity, not the implied disk activity) it's the meta partitions taking the pounding (ten second sample):</div><div><br class="webkit-block-placeholder"></div><div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> map addmap delmap getpag putpag pagio</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 45 0 0 /var/imap</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 11 11 11 17 0 0 /var/imap/meta-partition-1</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 290 290 290 463 5 0 /var/imap/meta-partition-2</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 139 139 139 183 3 0 /var/imap/meta-partition-3</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 66 66 66 106 10 0 /var/imap/meta-partition-7</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 347 347 342 454 16 0 /var/imap/meta-partition-8</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 57 57 57 65 5 0 /var/imap/meta-partition-9</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 4 4 8 4 0 0 /var/imap/partition-1</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 11 11 22 14 0 0 /var/imap/partition-2</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 1 1 2 1 0 0 /var/imap/partition-3</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 6 6 12 49 10 0 /var/imap/partition-7</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 15 15 28 457 0 0 /var/imap/partition-8</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 1 1 2 2 0 0 /var/imap/partition-9</span></font></font></div><div><br class="webkit-block-placeholder"></div><div>Similarly, by non-VM operation:</div><div><br class="webkit-block-placeholder"></div><div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> new name name attr attr lookup rddir read read write write</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> file remov chng get set ops ops ops bytes ops bytes</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 2.26K 0 6.15K 0 0 0 45 1.22K /var/imap</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 356 0 707 0 0 0 6 3.03K /var/imap/meta-partition-1</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 3 0 3 596 0 902 0 6 135K 90 305K /var/imap/meta-partition-2</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 621 0 1.08K 0 0 0 3 1.51K /var/imap/meta-partition-3</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 3 0 3 1.04K 0 1.70K 0 6 149K 36 650K /var/imap/meta-partition-7</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 2.28K 0 4.24K 0 0 0 7 1.87K /var/imap/meta-partition-8</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 18 0 32 0 0 0 2 176 /var/imap/meta-partition-9</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 2 2 2 22 0 30 0 1 2.37K 2 7.13K /var/imap/partition-1</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 3 4 12 84 0 157 0 1 677 3 7.51K /var/imap/partition-2</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 1 1 1 1.27K 0 2.16K 0 0 0 1 3.75K /var/imap/partition-3</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 2 2 4 35 0 56 0 1 3.97K 36 279K /var/imap/partition-7</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 1 2 1 256 0 514 0 0 0 1 3.75K /var/imap/partition-8</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> 0 0 0 0 0 0 0 0 0 0 0 /var/imap/partition-9</span></font></font></div><div><br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div>And looking at the real IO load, ten seconds of zpool (for the meta data and /var/imap_</div><div><br class="webkit-block-placeholder"></div><div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> capacity operations bandwidth</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">pool used avail read write read write</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">------------ ----- ----- ----- ----- ----- -----</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">pool1 51.6G 26.4G 0 142 54.3K 1001K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> mirror 25.8G 13.2G 0 68 38.4K 471K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> c0t0d0s4 - - 0 36 44.7K 471K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> c0t1d0s4 - - 0 36 0 471K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> mirror 25.8G 13.2G 0 73 15.9K 530K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> c0t2d0s4 - - 0 40 28.4K 531K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> c0t3d0s4 - - 0 39 6.39K 531K</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">------------ ----- ----- ----- ----- ----- -----</span></font></font></div><div><br class="webkit-block-placeholder"></div><div>is very different to ten seconds of sar for the NFS:</div><div><div><br class="webkit-block-placeholder"></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">09:46:34 device %busy avque r+w/s blks/s avwait avserv</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br class="webkit-block-placeholder"></span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">[...]</span></font></font></div><div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs73 1 0.0 3 173 0.0 4.2</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs86 3 0.1 12 673 0.0 6.5</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs87 0 0.0 0 0 0.0 0.0</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs89 0 0.0 0 0 0.0 0.0</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs96 0 0.0 0 0 0.0 1.8</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs101 1 0.0 1 25 0.0 8.0</span></font></font></div><div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;"> nfs102 0 0.0 0 4 0.0 9.4</span></font></font></div><div><br class="webkit-block-placeholder"></div><div>The machine has a _lot_ of memory (32GB) so it's likely that all mail that is delivered and then read within ten minutes never gets read back from the message store: the NFS load is almost entirely write as seen from the server.</div><div><br class="webkit-block-placeholder"></div><div>ian</div></div></div></div></div></div></body></html>