Industry Standard HTTP Server Benchmark Based on SPEC SFS (LADDIS)

I have converted LADDIS (industry standard SPEC benchmark to measure
NFS performance) to an HTTP server benchmark. Currently it uses a very
elementary workload (one URL). The results of this benchmark are
included at the end of this message.

The advantages of this benchmark are:
- it is a multiclient benchmark
- the client side implementation does not influence benchmark results
  which is the way it should be for a server benchmark
The disadvantages of this benchmark are:
- the current workload is not realistic

I need help to:
- Review benchmark methodology/implementation
- Create a realistic workload (or maybe different workloads for
  different environments)

LADDIS was originally developed by six vendors who saw the need for
better NFS benchmarks (Legato, Auspex, DEC, Data General, Interphase,
and Sun).  The original work group then took the benchmark to SPEC for
further development and promotion as an industry standard.  LADDIS has
considerably helped performance evaluation of NFS servers, thereby
contributing to the development of better servers.

I want to make the same thing happen for HTTP servers.  It's important
that such work not be done by any single vendor, nor with any single
narrow viewpoint of the requirements.  Would people in this group be
interested in working together to create an industry standard benchmark
for HTTP servers?

I would like to thank the SMCC Performance Engineering Group for their
support in this effort.

Regards,
Prasad

Note: this table illustrates the type of output generated by this
benchmark.  The actual numbers aren't meaningful due to the dummy
workload and uncontrolled test environment.

			Benchmark Results
************************************************************************

Aggregate Test Parameters:
    Number of processes = 1
    Requested Load (HTTP operations/second) = 10
    Warm-up time (seconds) = 1
    Run time (seconds) = 120
Aggregate Results for 1 Client(s), Tue Feb 28 13:37:34 1995
HTTP Server Benchmark Version 1, Creation - 15 February 1995
--------------------------------------------------------------------------
HTTP    Target Actual     HTTP   HTTP   Mean    Std Dev  Std Error   Pcnt
Op       HTTP   HTTP      Op     Op    Response Response of Mean,95%  of
Type     Mix    Mix     Success Error   Time     Time    Confidence  Total
         Pcnt   Pcnt     Count  Count  Msec/Op  Msec/Op  +- Msec/Op  Time
--------------------------------------------------------------------------
get        80%   82.0%        32     0  3271.62   822.29      9.93     85.9%
head       10%    5.1%         2     0  2450.00    28.69      7.42      4.0%
post       10%   12.8%         5     0  2437.80    43.81      5.80     10.0%
put         0%    0.0%         0     0     0.00     0.00      0.00      0.0%
delete      0%    0.0%         0     0     0.00     0.00      0.00      0.0%
checkout    0%    0.0%         0     0     0.00     0.00      0.00      0.0%
checkin     0%    0.0%         0     0     0.00     0.00      0.00      0.0%
showmethod  0%    0.0%         0     0     0.00     0.00      0.00      0.0%
link        0%    0.0%         0     0     0.00     0.00      0.00      0.0%
unlink      0%    0.0%         0     0     0.00     0.00      0.00      0.0%
--------------------------------------------------------------------------
INVALID RUN reported for Client 1 (haven).

        --------------------------------------------------------
        | AGGREGATE RESULTS SUMMARY                            | 
        --------------------------------------------------------
HTTP THROUGHPUT:       0 Ops/Sec   AVG. RESPONSE TIME:  3122.6 Msec/Op
HTTP MIXFILE: [ default ]
AGGREGATE REQUESTED LOAD: 10 Ops/Sec
TOTAL HTTP OPERATIONS:     39      TEST TIME: 119 Sec
NUMBER OF CLIENTS: 1

------------------------------------------------------------------------

************************************************************************

Received on Tuesday, 28 February 1995 20:35:39 UTC