Re: "Top N misconceptions about Web traffic that Internet implementers should know"

From: Pei Cao (
Date: Mon, May 24 1999

Date: Mon, 24 May 1999 12:54:35 -0500 (CDT)
From: Pei Cao <>
Message-Id: <>
Subject: Re: "Top N misconceptions about Web traffic that Internet implementers should know"

For your info, we were forced to turn off our client-side persistent
connection and our server-side persistent connection when we were testing
our proxy Peregrine in University of Wisconsin-Madison.  Here is what happened:

At the beginning, our proxy supports pipelining for both client side and
server side.  We scratched pipelining support later for two reasons:
1. none of the browser uses pipelining; 2. if none of the browsers uses
pipelining, the server's pipelining features have not been exercised, and
we don't want to be the one that debugs the Internet.  

When we started to do the testing, we find a really curious problem.  When
we maintain persistent connections with the browsers, we go to some site,
the browser simply doesn't display the content.  Meanwhile, the stop button
is on.  We click stop, the whole contents get displayed.  We turn off 
client-side persistent connection, the problem goes away.  Our guess: for 
some reason the browser simply doesn't display some contents until the 
connection has terminated.  

A second problem: many servers simply doesn't obey HTTP/1.1.  Examples
include: 304 replies carrying bodies, reply to HEAD request carrying bodies,
headers don't end in \r\n\r\n or even \n\n,  content-length is wrong, etc.
Just as Anja said, anything that could go wrong with HTTP/1.1, it goes wrong
at some server.  The complexity of HTTP/1.1 certainly didn't help in that
regard at all!  

It took us three weeks to figure out these problems and try to fix them.
The three weeks include quite a few all nighters.  In the end, we decided to 
simply turn off persistent connection.  Closing connection is a great way 
of flushing away the errors, and many bugs on today's Web servers are hidden
by closing connection (for example, content-length is wrong), and we simply 
don't want to be debugger for the Internet.

We would have loved to be completely HTTP/1.1 compliance.  We coded our
proxy that way.  We were forced to change because today's Internet simply
isn't ready for HTTP/1.1.

It doesn't matter how many Web servers are already HTTP/1.1 compliant.  All
it takes is the bottom 1% to force a proxy builder to be conservative.