Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers

Hi,

James Lacey wrote:

> Carl Kugler/Boulder/IBM wrote:
>
> > I stand corrected.  But then, why was Content-Length added in HTTP/1.0 and
> > Tranfer-Encoding: chunked in HTTP/1.1?
>
> Content-Length was added so that persistent connections would be possible.
>
> Without a content length, neither the client or the server would know
> where the message ended. You have to realize that TCP is a stream and
> it is up to the users of the stream to impose some kind of framing protocol
> onto the stream.
>
> Transfer-Encoding: chunked was added so that servers could return
> possibly large dynamic content in chunks, as it was generated, so that
> the server did not have to buffer up all of the dynamic content before
> sending it back. With chunking you can also have persistent connections
> w/o the need of a Content-Length header field.

"Transfer-Encoding chunked" also works on the client side, and would allow a
client to send dynamic content in its request. If both the client and the server
are doing chunking, it makes streaming applications possible over a standard HTTP
connection (eg: VPN, multimedia, etc).

I'd like to point out that with iPlanet Web Server 4.1, you can easily write such
applications in NSAPI and all the chunking issues are transparently taken care of
for the application developer. The only requirement is of course that the client
must be HTTP/1.1 compliant and support chunking. The client counterpart of such a
server-side streaming application could be a Java applet or browser plugin.

> One curious think about chunking that I do not understand is that
> HTTP proxies are required to move the chunked (any?) encoding before
> forwarding the response.

It seems that such a proxy would break the type of application mentioned above
... Full duplex would be required for such an application. This is something that
is not very well addressed in current HTTP specs (hint: this is just a trick to
get immediate attention from the RFC2616 owners :)).

Traditionally, with the content-length indicated in requests and responses, many
browsers and servers have used a synchronous model, where clients try to submit
the whole request, then only after it's complete, begin read the server response.
Many servers operate similarly and read the entire request before starting the
reponse. This is generally true for small GET requests, which the server can
afford to buffer. However, for POST, the server will not buffer since that would
make the memory footprint of the server very high when multiplied by the number
of connections. So the HTTP server will just act as a sort of intermediate and
use a relatively small buffer for the POST data, then pass it gradually to the
consumer of the request - whether it's a CGI, servlet, or NSAPI application ...

I have found that HTTP clients typically break if the server tries to respond too
early to a request, before it is complete. For example, this can happen with a
large POST request to a non-existing object. The server detects early on that the
URI doesn't point to a valid object on the server, and will try to respond
immediately with a 404 response. This is done before it reads the entire amount
of POST data ; since that can be very large and slow to read.

At that point however, the client is still submitting the large POST data ; and
the connection ends up being reset since both sides are sending and the client
isn't trying to read until its write loop is over. In iWS, we have some
workarounds that I consider ugly, including trying to read more POST data on such
erroneous large requests and just discarding it, as well as doing shutdown before
close, just to make sure the client doesn't see a "connection reset" error. This
doesn't work 100% of the time since we set some limits, but obviously we aren't
going to read megabytes of data if the request is too large, or wait a long time
on that socket, since that could be a denial of service attack - eating up socket
descriptors and in some cases also holding up a thread in the server. It would be
really good if the HTTP spec would address such cases and require both client and
servers to try both sides of the socket before closing/shutting the connection
down ... Such workarounds would not be necessary and the servers could just
peacefully shut down the connection after reporting an error, and assume that the
client in fact was really able to receive that error - not causing a "connection
reset".

--
The network is the computer, and it's down