Re: "Top N misconceptions about Web traffic that Internet implementers should know"

From: Martin F. Arlitt (arlitt@hpl.hp.com)
Date: Mon, May 24 1999


Message-ID: <37498998.302D65FC@hpl.hp.com>
Date: Mon, 24 May 1999 17:17:12 +0000
From: "Martin F. Arlitt" <arlitt@hpl.hp.com>
To: www-wca@w3.org
Subject: Re: "Top N misconceptions about Web traffic that Internet   implementers   should know"

I did I quick analysis of 200 "popular" sites (I got these from a
MediaMetrix list) and got the following results:

1/3 of the servers replied with "HTTP/1.0 <status code>"
2/3 of the servers replied with "HTTP/1.1 <status code>"

Of the "HTTP/1.1" servers, only 1/2 supported persistent connections -
1/3 of all the servers tested.

However, during the testing I noticed that a lot of the servers appeared
to be "unfriendly" towards crawlers and would terminate connections
abruptly in some cases.  This may influence the results of the number of
servers that have persistent connections turned on.  I also noticed that
some of the sites that utilize multiple servers have different servers
running, which makes the results difficult to repeat.

While I am not sure how accurate these results are, I would not be
surprised if persistent connections are not as widely supported as some
people might think.  I base this in part on discussions I sat in on with
a number of system administrators - they stated that they explicitly
turned off persistent connections.

If there is interest within the group I am willing to redo this work in a
more thorough manner, or participate with others on it.

Martin