stress testing

[Rob Siemborski]

On 18 Sep 2002, Hein Roehrig wrote:

> Before going production with a new server I would like to do some stress
> testing esp wrt locking problems. A quick search of the archives and at
> Google shows that people have been looking for scripts etc. but there is
> no standard open-source test/stress suite and nobody so far posted their
> scripts.

This isn't really surprising when you consider just how hard it is to
create a reasonable simulation of IMAP traffic.  Yes, you can create
scripts relatively easily that just repeat commands or similar-looking
sequences of commands, but to get something that looks like actual traffic
is relatively hard.

Keep in mind that you can't just launch thousands of connections at a
server, you need to space them out over a reasonable time period since in
reality, the server will never be slammed with several thousand
connections within a short time period.  Additionally, you probably want
to spread the load out between various partitions in a reasonable way that
simulates your actual use.

If you want to verify all of the responses, it becomes even harder.

> Before I start creating my own scripts, I would welcome pointers to work
> I may have missed and/or advice. For instance, there is a claim in
> comp.mail.imap message <63bde6$krb at engnews1.Eng.Sun.COM> that a perl
> script will not be fast enough to really stress a decent server... I
> find that hard to believe.

Unfortuinately, this confirms our experience.  Generally what we do is we
(slowly) start a large number of slightly modified imtest processes that
read in various lists of IMAP commands with a time to wait between each
(this is the slight modification).  Then we just watch what happens as the
scripts interact with the server.

Historically, we've needed to spread the load generation out between
multiple client machines, since the clients *do* tend to die under the
load long before the server does.  Of course, the clients generally have
less memory than the server, and the server is (hopefully) optimized to
handle high loads, where there hasn't been much work towards optimizing
imtest ;)

I can only expect that a perl script would do worse, since it has all the
overhead of perl along with the script itself.

-Rob

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456
Research Systems Programmer * /usr/contributed Gatekeeper