Re: Last-Modified in chunked footer

I've been thinking about this issue for a few days.  My original
message, discussing just the effects on proxies, was clearly
insufficient; there are many user agents that want to see some
of the headers before the entire message has arrived, so that
(for example) they can start rendering HTML as it arrives.

But I think we are running into a basic confusion between
what is "necessary" and what is "fast".

The original motivation for trailers was to allow a server
that was generating content on the fly to send a header
(e.g., Content-MD5) that could not be generated before the
entire content.

Similarly, the desire of a browser to see, say, Content-type
at the beginning of a message is so that it can render the
message "on the fly".

And the analysis that I sent a few days ago looked at
what a proxy would need to know in order to forward a response
"on the fly", without having to first buffer it.

Clearly, if a response is based on a static value, it seems
preferrable (and easiest) if the server puts the headers before
the content.  The trickier question is for dynamically-generated
responses.

In many (most) cases, any implementation (server, proxy, or client)
could, in principle, buffer the entire message so that it
would not be handled on the fly.  While this would certainly
add latency (perhaps unacceptable latency), it would not force
incorrect behavior.  (This assumes that the client *knows* that
it needs to buffer the message, which is a separate issue).

In some cases, however (such as an indefinitely long server push),
it is actually impossible to buffer the entire response at any
point.  In such cases, one clearly needs to put the headers
before the content!  This would seem to preclude the use of
Content-MD5 (and some forms of Digest Authentication?) in these
cases.

In all of the other cases of responses "generated on the fly",
however, the question seems to be "who gets stuck having to buffer
the entire message?"  If there are any header fields whose value
depends on the value of the body, one end (at least) is going
to have to do this.  It could either be the sender or the receiver;
which one makes more sense?

>From a "correctness" point of view, the answer is "neither, if
the recipient knows to expect trailers."  From an efficiency
point of view, however, I think it's better to have the sender
do the buffering, in most cases.

The reason for this is, except in rare cases, it takes
less time to generate the response at the server than it
takes to transmit it over the network.  Therefore, buffering
at the server means adding a relatively short delay before
the client can start rendering the content.  Buffering at
the client, on the other hand, means that one has to pay
for the entire network latency for the entire response before
any rendering can take place.

Thus, in general, I would expect that server designers who
want to minimize "user perceived latency" would be willing
to buffer their non-infinite responses, if this is what it
takes to add content-dependent headers in front of the content.

However, this does not mean that we should force a server
to buffer responses.  It may not be possible (the server
may have limited resources, or the response may be indefinitely
long).  This means that clients ought to have the ability
to correctly (if belatedly!) render any response with any
content-dependent header in the trailer.  (People have pointed
out that the Vary header might possibly be content-dependent,
but this header should never affect rendering.)

What this boils down to is a version of the Robustness
Principle ("be conservative in what you send, be liberal
in what you accept").

I would propose that the spec make what appears to be
a contradictory set of requirements:

	(1) recipients MUST be able to deal with any
	content-dependent header in the trailer of a chunked
	encoding. 

	(2) senders SHOULD NOT put any headers in the
	trailer unless it is impossible to put the
	headers before the content, or unless the
	cost of putting the header before the content
	is larger than the expected network latency
	for transmitting the entire response.

	(3) The set of content-dependent headers may
	be extended in the future, but currently includes
		Content-*
		Expires
		Etag
		Vary
		Last-Modified
		Authorization
		AuthenticationInfo
	(and probably a few others that I'm missing).		

It does seem reasonable to introduce a "hint" (i.e., non-mandatory)
response-header, as suggested by both Maurizio Codogno and by
Klaus wide, that tells the recipient "look in the trailer before
rendering" (as in the Postscript "(atend)" comment).  Since I don't
think we can successfully insist that all clients MUST delay rendering
if they see such a header, it can only be a "hint", but at least
it gives the client a chance to decide (based on user preferences?)
whether to be cautious or low-latency.

-Jeff

Received on Friday, 5 September 1997 16:32:31 UTC