Content negotiation and Vary (1.3, 12, 13.5, 14.43)

I just spent half a day reviewing the changed negotiation and vary
stuff in pre-04, and almost lost my mind thinking through the
interaction between variance due to transparent negotiation, variance
due to negotiated error responses, and the areas left unspecified by
the spec.

I am relieved to report that of the various jarring near-truths,
subtle internal contradictions, and omitted rulings on interaction
with other protocol features I found, none seem to be serious enough
to threaten interoperability if people start implementing this stuff.

In the new pre-04 draft, the term `content negotiation' not only used
for a) the selection of the representation _of_ a resource to be
included in a successful response, as it was in the 00, 01, 02 and 03
drafts, but also for b) the selection of the representation of an
(error) message _about_ a resource.

I think a) and b) are orthogonal and should be covered as such.
However, I don't feel strong enough about this to make a big fuss
about having the terminology and scope of the Vary header changed this
late in the editing process.  For the record, though b) is a form of
negotiation, it should not be called `content negotiation'.  It should
be called something like `error message negotiation'.  b) has nothing
to do with agent-driven and transparent negotiation as covered in
Section 12.

I do want to suggest various editorial changes to the content
negotiation text.  The text is _very_ confusing in some places.  Even
though it is, as far as I can tell, not confusing in a way that
threatens interoperability, it may be confusing in a way that causes
trouble in a review.

>1.3 Terminology
>[...]
>representation
>
>  An entity included with a response that is subject to content
>  negotiation, as described in section 12.  There may exist multiple
>  representations associated with a particular response status of a
>  requested resource.
>
>content negotiation
>  The mechanism for selecting the appropriate representation when
>  servicing a request, as described in section 12.  Note that response
>  messages may be negotiated, as well as resources.

This last sentence is confusing.  I suggest that this sentence is
deleted.

>
>variant

The problems with this one have already been covered on the list.

[...]

>12 Content Negotiation
>
>Most HTTP responses include an entity which contains information for
>interpretation by a human user. Naturally, it is desirable to supply the
>user with the "best available" entity corresponding to the request.
>Unfortunately for servers and caches, not all users have the same
>preferences for what is "best," and not all user agents are equally
>capable of rendering all entity types. For that reason, HTTP has
>provisions for several mechanisms for "content negotiation" -- the
>process of selecting the best representation of a resource for a given
                                                                  ^^^^^ 
>response when there are multiple representations available.
 ^^^^^^^^

This `given response' is not accurate, because content negotiation is
also sometimes involved in the decision _which_ response (200, 300 or
406) is `the given response'.  However, writing a complete description
of the interaction between content negotiation and the choice of the
response status code is not feasible anymore at this point.  As the
rules for the treatment of vary headers by caches allow the
interaction to be present, it does not hurt that much to imply that
the interaction is not present.  So I suggest to keep the above text
as it is.

>  Note: This is not called "format negotiation" because the alternate
>  representations may be of the same media type, but use different
>  capabilities of that type, be in different languages, etc.
>
>Any response containing an entity-body MAY be subject to negotiation,
>including error responses.
>
>There are two kinds of content negotiation which are possible in HTTP:
>server-driven and agent-driven negotiation. These two kinds of
>negotiation are orthogonal and thus may be used separately or in
>combination without affecting the interpretation of a response. One
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This seems to imply that other things _do_ affect the interpretation
of a response, but I can't think of any such things being discussed in
the protocol.  I suggest that the marked text above is deleted.

>method of combination, referred to as transparent negotiation, occurs
>when a cache uses the agent-driven negotiation information provided by
>the origin server in order to provide server-driven negotiation for
>subsequent requests.
>
>
>12.1 Server-driven Negotiation
>
>If the selection of the best representation for a response is made by an
>algorithm located at the origin server, it is called server-driven
                          ^^^^^^
Delete `origin': the last paragraph of the previous section is
talking about a cache also doing server-driven negotiation.  Also, a
proxy selecting the language for the text in a 504 error message would
be doing server-driven negotiation.  So a limitation to origin servers
is clearly wrong.

>negotiation.  Selection is based on the available representations of the
>response (the dimensions over which it can vary) and the contents of
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the first use of the word `dimensions' in the spec.  The
parenthetical remark seems to imply that a dimension is the same as a
representation, but this is clearly not how `dimension' is being used
later. This discrepancy could lead to lots of confusion for an
unsuspecting reader.  I suggest that the parenthetical remark is
removed.
  
>particular header fields in the request message or on other information
>pertaining to the request (such as the network address of the client).
>
>Server-driven negotiation is advantageous when the algorithm for
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I suggest that the start of this paragraph is rewritten so that
negotiation of error responses is also covered.  New text:

 Server-driven negotiation can be used for selecting the
 representation of an error response, and also for selecting the
 representation for a succesfull response.  When selecting the
 representation to be used in a successful response, server-driven
 negotiation is advantageous when the algorithm for ...

>selecting from among the available representations is difficult to
>describe to the user agent, or when the server desires to send its "best
>guess" to the client along with the first response (hoping to avoid the
>round-trip delay of a subsequent request if the "best guess" is good
>enough for the user).  In order to improve the server's guess, the user
>agent MAY include request header fields (Accept, Accept-Language,
>Accept-Encoding, etc.) which describe its preferences for such a
>response.
>
>Server-driven negotiation has disadvantages:
>
>1.   It is impossible for the server to accurately determine what might be
>  "best" for any given user, since that would require complete
>  knowledge of both the capabilities of the user agent and the intended
>  use for the response (e.g., does the user want to view it on screen
>  or print it on paper?).
>
>2.   Having the user describe its capabilities in every request can be
                 ^^^^ 
`user' must by `user agent'.

>  both very inefficient (given that only a small percentage of
>  responses have multiple representations) and a potential violation of
>  the user's privacy.
>
>3.   It significantly complicates the implementation of an origin server
         ^^^^^^^^^^^^^
>  and the algorithms for generating responses to a request.

I think that `significantly' should be removed: even if it were true
that there would be significantly more complicated, we would not want
to say it.

>
>4.   It may interfere with a public cache's ability to use the same
             ^^^^^^^^^^^^^^
>  response for multiple user's requests.

I think that `interfere with' suggests that a public cache may, if it
is smart enough, be able to route around server driven negotiation.
This fatal suggestion must be avoided.  I suggest replacing `interfere
with' with `limit'.

>
>HTTP/1.1 includes the following request-header fields for enabling
>server-driven negotiation through description of user agent capabilities
>and user preferences: Accept (section 14.1), Accept-Charset (section
>14.2), Accept-Encoding (section 14.3), Accept-Language (section 14.4),
>and User-Agent (section 14.42). However, an origin server is not limited
>to these dimensions and MAY vary the response based on any aspect of the
>request, including information outside the request-header fields or
>within extension header fields not defined by this specification.
>
>HTTP/1.1 origin servers MUST include an appropriate Vary header field
          ^^^^^^
Delete `origin', as discussed above.

>(section 14.43) in any response based on server-driven negotiation. The
                      ^^^
insert `cachable', as discussed in earlier messages.

>Vary header field describes the dimensions over which the response might
>vary (i.e., the dimensions over which the origin server picks its "best
                                           ^^^^^^
Delete `origin', as discussed above.

>guess" response from multiple representations).
>
>HTTP/1.1 public caches MUST recognize the Vary header field when it is
>included in a response and obey the requirements described in section
>13.5 that describes the interactions between caching and content
>negotiation.
>
>
>12.2 Agent-driven Negotiation
>
>With agent-driven negotiation, selection of the best representation for
>a response is performed by the user agent after receiving an initial
>response from the origin server. Selection is based on a list of the
>available representations of the response included within the header
>fields (this specification reserves the keyword Alternates, to be
>defined in a separate specification[27]) or entity-body of the initial
>response, with each representation identified by its own URI. Selection
>from among the representations may be performed automatically (if the
>user agent is capable of doing so) or manually by the user selecting
>from a generated (possibly hypertext) menu.
>
>Agent-driven negotiation is advantageous when the response would vary
>over commonly-used dimensions (such as type, language, or encoding),
>when the origin server is unable to determine a user agent's
>capabilities from examining the request, and generally when public
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^
>caches are used to distribute server load and reduce network usage.
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I don't understand this last claim.  Caches can optimize both
agent-driven and server-driven negotiation, but neither agent-driven
nor server-driven optimize caching.  I suggest that this claim is
either removed or that an explanation of this claim is added.

>
>Agent-driven negotiation suffers from the disadvantage of needing a
>second request to obtain the best alternate representation.  This second
                                                              ^^^^^^^^^^^
>request is only efficient when caching is used.
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

`is only efficient when caching is used' is a predicate that is
applicable to most requests, but not to all second requests in
agent-driven negotiation.  If the agent and server run on the same
internet host, use of a cache will not make the second request more
efficient.  This sentence makes a false claim, and should be deleted.

>                                                In addition, this
>specification does not define any mechanism for supporting automatic
>selection, though it also does not prevent any such mechanism from being
>developed as an extension and used within HTTP/1.1.
>
>HTTP/1.1 defines the 300 (Multiple Choices) and 406 (Not Acceptable)
>status codes for enabling agent-driven negotiation when the server is
>unwilling or unable to provide a varying response using server-driven
>negotiation.
>
>
>12.3 Transparent Negotiation

Let me repeat my earlier objection to this section: there is no reason
for including this section, as a discussion of transparent negotiation
is not needed to justify the HTTP/1.1 protocol elements defined in the
pre-04 spec.  The inclusion of this section will only increase the
risk that the pre-04 spec is rejected on external review and might
unleash a flood of questions on the mailing list that somebody would
have to answer.

I suggest that this section is removed, and if it is not removed, I
ask that the suggested improvements below are made.

>Transparent negotiation is a combination of both server-driven and
>agent-driven negotiation.  When a cache is supplied with an automated
                                                          ^^^^^^^^^^^^
>form of the list of available representations of the response (as in
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>agent-driven negotiation) and the dimensions of variance are completely
 ^^^^^^^^^^^^^^^^^^^^^^^^^

I don't know what an `automated form of a list' is.  I suggest
rewriting this to 

 a form of the list of available representations of the response (as in
 agent-driven negotiation) which supports automatic selection,

>understood by the cache, then the cache becomes capable of performing
>server-driven negotiation on behalf of the origin server for subsequent
>requests on that resource.
>
>Transparent negotiation has the advantage of distributing the
>negotiation work that would otherwise be required of the origin server
>and also removing the second request delay of agent-driven negotiation
>when the cache is able to correctly guess the right response and already
                                                              ^^^^^^^^^^^ 
>has that response cached.
 ^^^^^^^^^^^^^^^^^^^^^^^^^

In this case, the cache even removes the second request if it does not
have the response cached, so the marked text above should be removed.

>
>A cache performing transparent negotiation MUST include the agent-driven
>negotiation information along with the response, and MUST add a Vary
>header field to the response (defining the dimensions of its variance)
>if a Vary field was not already assigned by the origin server.
>
>These requirements apply to HTTP/1.1 applications even though this
>specification does not specify transparent negotiation, since an
>understanding of these requirements is a necessary prerequisite for any
                                                    ^^^^^^^^^^^^  
>future implementation of these features.

I read this to mean that HTTP/1.1 applications must be ready to
interoperate with _future_ HTTP/1.1 caches which implement transparent
negotiation, rather than the other way around.

In my reading, this part of the pre-04 text makes normative reference
to a standard that has yet to be defined, which is clearly something
that cannot be allowed.

I suggest a complete rewrite of the last two paragraphs of 12.3 to:

 This specification does not define any mechanism for transparent
 negotiation, though it also does not prevent any such mechanism from
 being developed as an extension and used within HTTP/1.1.  A HTTP/1.1
 cache performing transparent negotiation MUST include a Vary header
 field in the response (defining the dimensions of its variance) to
 ensure correct interoperation with all HTTP/1.1 clients.


Koen.


-------------------------------------------------------------
	            DE  DIGITALE  STAD

Received on Sunday, 2 June 1996 06:51:06 UTC