improving concurrency/performance

[John Madden]

> It's situations like this Dtrace was made for. But on linux we still have
> to use some 'gut feeling' to figure it out ...

True.  It's that sort of tool that I'm looking for, specifically to look into
concurrency on the skiplist db's, as the system load is so low that it seems
there's got to be a simple explanation for what's going on.

> So you say you have fast disks for bonnie, but still see slow imap copy
> operations? What kind of SAN exactly do you have attached? Because fsync()
> calls would still be my primary suspect here ...

It's an IBM DS-6800.

> You say you're copying mail spool from one box to another via imap. Is the
> source box able to provide mails at a fast enough rate?

Yeah, the load average there dropped to near zero, with no iowait and only 1-3MB/s
coming off its disk.

Perhaps it's worth repeating: With a single imapcopy process, the whole thing goes
along pretty quickly, but drops off significantly with a second process and comes
to basically a crawl with just 5 processes running concurrently.  I gambled that I
could shorten my migration by running more than one at a time since one only seems
to raise the load on the box to 0.80.  With 5, I'm only able to get it to around
2.5 and only briefly as the throughput starts to drop off.

With a little multi-threaded perl, I wrote a quick benchmark script that (in
parallel) grabs a random user, logs in, selects the INBOX, pulls out the first
message and logs out.  I'm able to do about 230 of those per second, so at least
the read performance is more than acceptable.  (And the client box here, a 4-CPU
Opteron 850, is definitely the bottleneck anyway.)

John



-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden at ivytech.edu