Content negotiation and Vary (1.3, 12, 13.5, 14.43)

>Date:  Sun, 2 Jun 1996 14:12:45 +0200 (MET DST)
>From:  koen holtman <>
>Subject:  Content negotiation and Vary (1.3, 12, 13.5, 14.43)
>I just spent half a day reviewing the changed negotiation and vary
>stuff in pre-04, and almost lost my mind thinking through the
>interaction between variance due to transparent negotiation, variance
>due to negotiated error responses, and the areas left unspecified by
>the spec.
>I am relieved to report that of the various jarring near-truths,
>subtle internal contradictions, and omitted rulings on interaction
>with other protocol features I found, none seem to be serious enough
>to threaten interoperability if people start implementing this stuff.
>In the new pre-04 draft, the term `content negotiation' not only used
>for a) the selection of the representation _of_ a resource to be
>included in a successful response, as it was in the 00, 01, 02 and 03
>drafts, but also for b) the selection of the representation of an
>(error) message _about_ a resource.

HTTP transfers entities; some come from resources, some contain
information from servers the state of requests (often errors,
or other useful information for end users.  It makes no
sense in terms of the protocol to have to treat entities
differently from each other.

The definitions were modified to make it clear that content
negotiation applies to any response to a request made by a client;
to do otherwise would just complicate the system.

The mental shift one has to get away from is thinking about
Accept* just applying to a resource.  They can just as well
apply to any message or entity returned in a request.

>I think a) and b) are orthogonal and should be covered as such.
>However, I don't feel strong enough about this to make a big fuss
>about having the terminology and scope of the Vary header changed this
>late in the editing process.  For the record, though b) is a form of
>negotiation, it should not be called `content negotiation'.  It should
>be called something like `error message negotiation'.  b) has nothing
>to do with agent-driven and transparent negotiation as covered in
>Section 12.
Sure it does; it even uses the same mechanisms.

>I do want to suggest various editorial changes to the content
>negotiation text.  The text is _very_ confusing in some places.  Even
>though it is, as far as I can tell, not confusing in a way that
>threatens interoperability, it may be confusing in a way that causes
>trouble in a review.
>>1.3 Terminology
>>  An entity included with a response that is subject to content
>>  negotiation, as described in section 12.  There may exist multiple
>>  representations associated with a particular response status of a
>>  requested resource.
>>content negotiation
>>  The mechanism for selecting the appropriate representation when
>>  servicing a request, as described in section 12.  Note that response
>>  messages may be negotiated, as well as resources.
>This last sentence is confusing.  I suggest that this sentence is

I think the fact that responses in general are negotiable is
worty of note.  See if this sentence does better

Note, therefore that responses of all sorts can be negotiated
(including error responses), as well as entities derived from

>The problems with this one have already been covered on the list.

I tried living without it entirely and failed.  The verbiage
gets too clumsy, in my opinion.  I reworked
its definition to make sure that the single case is possible.
A different word would be nice, but we've been unable to come up
with one.

>>12 Content Negotiation
>>Most HTTP responses include an entity which contains information for
>>interpretation by a human user. Naturally, it is desirable to supply the
>>user with the "best available" entity corresponding to the request.
>>Unfortunately for servers and caches, not all users have the same
>>preferences for what is "best," and not all user agents are equally
>>capable of rendering all entity types. For that reason, HTTP has
>>provisions for several mechanisms for "content negotiation" -- the
>>process of selecting the best representation of a resource for a given
>                                                                  ^^^^^ 
>>response when there are multiple representations available.
> ^^^^^^^^
>This `given response' is not accurate, because content negotiation is
>also sometimes involved in the decision _which_ response (200, 300 or
>406) is `the given response'.  However, writing a complete description
>of the interaction between content negotiation and the choice of the
>response status code is not feasible anymore at this point.  As the
>rules for the treatment of vary headers by caches allow the
>interaction to be present, it does not hurt that much to imply that
>the interaction is not present.  So I suggest to keep the above text
>as it is.
>>  Note: This is not called "format negotiation" because the alternate
>>  representations may be of the same media type, but use different
>>  capabilities of that type, be in different languages, etc.
>>Any response containing an entity-body MAY be subject to negotiation,
>>including error responses.
>>There are two kinds of content negotiation which are possible in HTTP:
>>server-driven and agent-driven negotiation. These two kinds of
>>negotiation are orthogonal and thus may be used separately or in
>>combination without affecting the interpretation of a response. One
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>This seems to imply that other things _do_ affect the interpretation
>of a response, but I can't think of any such things being discussed in
>the protocol.  I suggest that the marked text above is deleted.

I think you are right.  I've deleted the phrase (which I can't figure
out what it was referring to either).

>>method of combination, referred to as transparent negotiation, occurs
>>when a cache uses the agent-driven negotiation information provided by
>>the origin server in order to provide server-driven negotiation for
>>subsequent requests.
>>12.1 Server-driven Negotiation
>>If the selection of the best representation for a response is made by an
>>algorithm located at the origin server, it is called server-driven
>                          ^^^^^^
>Delete `origin': the last paragraph of the previous section is
>talking about a cache also doing server-driven negotiation.  Also, a
>proxy selecting the language for the text in a 504 error message would
>be doing server-driven negotiation.  So a limitation to origin servers
>is clearly wrong.

I think you are right; I've deleted origin.

>>negotiation.  Selection is based on the available representations of the
>>response (the dimensions over which it can vary) and the contents of
>          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>This is the first use of the word `dimensions' in the spec.  The
>parenthetical remark seems to imply that a dimension is the same as a
>representation, but this is clearly not how `dimension' is being used
>later. This discrepancy could lead to lots of confusion for an
>unsuspecting reader.  I suggest that the parenthetical remark is

Better, I think, to expand slightly.  I've added
"(the dimensions over which it can vary; e.g. language, content-coding, etc.)"
>>particular header fields in the request message or on other information
>>pertaining to the request (such as the network address of the client).
>>Server-driven negotiation is advantageous when the algorithm for
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>I suggest that the start of this paragraph is rewritten so that
>negotiation of error responses is also covered.  New text:
> Server-driven negotiation can be used for selecting the
> representation of an error response, and also for selecting the
> representation for a succesfull response.  When selecting the
> representation to be used in a successful response, server-driven
> negotiation is advantageous when the algorithm for ...

The real bug is the definition of representation, which I don't think
Roy dealt with properly.  I've reworked the representation definition
to make clear we mean any entity, not just those of resources.

  An entity included with a response that is subject to content
  negotiation, as described in section 12.  There may exist multiple
  representations associated with a particular resource.  Likewise,
  there may be multiple representation of entities returned in error

>>selecting from among the available representations is difficult to
>>describe to the user agent, or when the server desires to send its "best
>>guess" to the client along with the first response (hoping to avoid the
>>round-trip delay of a subsequent request if the "best guess" is good
>>enough for the user).  In order to improve the server's guess, the user
>>agent MAY include request header fields (Accept, Accept-Language,
>>Accept-Encoding, etc.) which describe its preferences for such a
>>Server-driven negotiation has disadvantages:
>>1.   It is impossible for the server to accurately determine what might be
>>  "best" for any given user, since that would require complete
>>  knowledge of both the capabilities of the user agent and the intended
>>  use for the response (e.g., does the user want to view it on screen
>>  or print it on paper?).
>>2.   Having the user describe its capabilities in every request can be
>                 ^^^^ 
>`user' must by `user agent'.



>>  both very inefficient (given that only a small percentage of
>>  responses have multiple representations) and a potential violation of
>>  the user's privacy.
>>3.   It significantly complicates the implementation of an origin server
>         ^^^^^^^^^^^^^
>>  and the algorithms for generating responses to a request.
>I think that `significantly' should be removed: even if it were true
>that there would be significantly more complicated, we would not want
>to say it.
>>4.   It may interfere with a public cache's ability to use the same
>             ^^^^^^^^^^^^^^
>>  response for multiple user's requests.
>I think that `interfere with' suggests that a public cache may, if it
>is smart enough, be able to route around server driven negotiation.
>This fatal suggestion must be avoided.  I suggest replacing `interfere
>with' with `limit'.


>>HTTP/1.1 includes the following request-header fields for enabling
>>server-driven negotiation through description of user agent capabilities
>>and user preferences: Accept (section 14.1), Accept-Charset (section
>>14.2), Accept-Encoding (section 14.3), Accept-Language (section 14.4),
>>and User-Agent (section 14.42). However, an origin server is not limited
>>to these dimensions and MAY vary the response based on any aspect of the
>>request, including information outside the request-header fields or
>>within extension header fields not defined by this specification.
>>HTTP/1.1 origin servers MUST include an appropriate Vary header field
>          ^^^^^^
>Delete `origin', as discussed above.

Aren't Vary headers passed through?  I don't think that caches
should add them by themselves; I believe Roy is right here.

>>(section 14.43) in any response based on server-driven negotiation. The
>                      ^^^
>insert `cachable', as discussed in earlier messages.

Negotiation needs to be signalled on all responses, not just
cacheable ones.  I think it correct as written.
>>Vary header field describes the dimensions over which the response might
>>vary (i.e., the dimensions over which the origin server picks its "best
>                                           ^^^^^^
>Delete `origin', as discussed above.

See above.

>>guess" response from multiple representations).
>>HTTP/1.1 public caches MUST recognize the Vary header field when it is
>>included in a response and obey the requirements described in section
>>13.5 that describes the interactions between caching and content
>>12.2 Agent-driven Negotiation
>>With agent-driven negotiation, selection of the best representation for
>>a response is performed by the user agent after receiving an initial
>>response from the origin server. Selection is based on a list of the
>>available representations of the response included within the header
>>fields (this specification reserves the keyword Alternates, to be
>>defined in a separate specification[27]) or entity-body of the initial
>>response, with each representation identified by its own URI. Selection
>>from among the representations may be performed automatically (if the
>>user agent is capable of doing so) or manually by the user selecting
>>from a generated (possibly hypertext) menu.
>>Agent-driven negotiation is advantageous when the response would vary
>>over commonly-used dimensions (such as type, language, or encoding),
>>when the origin server is unable to determine a user agent's
>>capabilities from examining the request, and generally when public
>                                          ^^^^^^^^^^^^^^^^^^^^^^^^^
>>caches are used to distribute server load and reduce network usage.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>I don't understand this last claim.  Caches can optimize both
>agent-driven and server-driven negotiation, but neither agent-driven
>nor server-driven optimize caching.  I suggest that this claim is
>either removed or that an explanation of this claim is added.

Dunno.  It is 1am, and I don't have time to think about it.

>>Agent-driven negotiation suffers from the disadvantage of needing a
>>second request to obtain the best alternate representation.  This second
>                                                              ^^^^^^^^^^^
>>request is only efficient when caching is used.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>`is only efficient when caching is used' is a predicate that is
>applicable to most requests, but not to all second requests in
>agent-driven negotiation.  If the agent and server run on the same
>internet host, use of a cache will not make the second request more
>efficient.  This sentence makes a false claim, and should be deleted.

Agent negotiation isn't significantly less efficient for the Internet
as a whole than server-driven, if there is a local cache to service
the request so that you don't have two complete round trips to
the origin server.  So in fact, it does depend on the second
request.  My brain is getting too tired to say this better right now,
and I'm not going to worry about it.

>>                                                In addition, this
>>specification does not define any mechanism for supporting automatic
>>selection, though it also does not prevent any such mechanism from being
>>developed as an extension and used within HTTP/1.1.
>>HTTP/1.1 defines the 300 (Multiple Choices) and 406 (Not Acceptable)
>>status codes for enabling agent-driven negotiation when the server is
>>unwilling or unable to provide a varying response using server-driven
>>12.3 Transparent Negotiation
>Let me repeat my earlier objection to this section: there is no reason
>for including this section, as a discussion of transparent negotiation
>is not needed to justify the HTTP/1.1 protocol elements defined in the
>pre-04 spec.  The inclusion of this section will only increase the
>risk that the pre-04 spec is rejected on external review and might
>unleash a flood of questions on the mailing list that somebody would
>have to answer.

I think it very likely the spec will be rejected on this point.
I think people have to see some idea of where we are going,
to understand the requirements we've laid on them.
>I suggest that this section is removed, and if it is not removed, I
>ask that the suggested improvements below are made.
>>Transparent negotiation is a combination of both server-driven and
>>agent-driven negotiation.  When a cache is supplied with an automated
>                                                          ^^^^^^^^^^^^
>>form of the list of available representations of the response (as in
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>agent-driven negotiation) and the dimensions of variance are completely
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>I don't know what an `automated form of a list' is.  I suggest
>rewriting this to 
> a form of the list of available representations of the response (as in
> agent-driven negotiation) which supports automatic selection,

>>understood by the cache, then the cache becomes capable of performing
>>server-driven negotiation on behalf of the origin server for subsequent
>>requests on that resource.
>>Transparent negotiation has the advantage of distributing the
>>negotiation work that would otherwise be required of the origin server
>>and also removing the second request delay of agent-driven negotiation
>>when the cache is able to correctly guess the right response and already
>                                                              ^^^^^^^^^^^ 
>>has that response cached.
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>In this case, the cache even removes the second request if it does not
>have the response cached, so the marked text above should be removed.

How could the cache help if it doesn't have that response cached?
Removing the phrase seems to me to reduce the sentence to useless.

>>A cache performing transparent negotiation MUST include the agent-driven
>>negotiation information along with the response, and MUST add a Vary
>>header field to the response (defining the dimensions of its variance)
>>if a Vary field was not already assigned by the origin server.
>>These requirements apply to HTTP/1.1 applications even though this
>>specification does not specify transparent negotiation, since an
>>understanding of these requirements is a necessary prerequisite for any
>                                                    ^^^^^^^^^^^^  
>>future implementation of these features.
>I read this to mean that HTTP/1.1 applications must be ready to
>interoperate with _future_ HTTP/1.1 caches which implement transparent
>negotiation, rather than the other way around.
>In my reading, this part of the pre-04 text makes normative reference
>to a standard that has yet to be defined, which is clearly something
>that cannot be allowed.
>I suggest a complete rewrite of the last two paragraphs of 12.3 to:
> This specification does not define any mechanism for transparent
> negotiation, though it also does not prevent any such mechanism from
> being developed as an extension and used within HTTP/1.1.  A HTTP/1.1
> cache performing transparent negotiation MUST include a Vary header
> field in the response (defining the dimensions of its variance) to
> ensure correct interoperation with all HTTP/1.1 clients.

I see what you are driving at, and talked with Roy about it.
This is what I came up with afterwards:

The following requirements apply to HTTP/1.1 applications even though
this specification does not specify transparent negotiation, to ensure
that a transparent negotiation extension can be added at a later time
and used within HTTP/1.1. Any cache that performs transparent
negotiation SHOULD include the agent-driven negotiation information
along with the response, and MUST add a Vary header field to the
response (defining the dimensions of its variance) if a Vary field was
not already assigned by the origin server, to ensure correct operation
with all HTTP/1.1 clients.

Maybe a bit verbose, but it gets the point across.  

			Thanks for the comments.
				- Jim

Received on Sunday, 2 June 1996 22:43:34 UTC