<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Hi All,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">We were debugging the CPU usage in a ctl_conversationsdb rebuild yesterday, and noticed an interesting thing. 70% of the CPU utilisation for this one process was inside the kernel! Mostly with dirty pages.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">ctl_conversationsdb -R is particularly heavy on the twoskip database - it's rewriting a lot of random keys. This leads to writes all over the place, as it stitches records into the skiplists.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Of course the "real answer"[tm] is zeroskip, which doesn't do random writes - but until then, we suspect that the cost is largely due to the face that we use mmap to read, and fwrite to write! We know that might be less efficient already from Linus' comments about 10 years ago! And I guess here's the proof.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">An option would be to switch to using mmap to write as well. We could easily modify lib/mappedfile to memcpy to do the writes.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Does anybody see any strong reason not to?<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>