Re: Last Call: HTTP Extension Framework to Proposed Standard

From: Henrik Frystyk Nielsen (
Date: Thu, Feb 11 1999

Message-Id: <>
Date: Thu, 11 Feb 1999 14:24:09 -0500
To: Koen Holtman <>,
From: Henrik Frystyk Nielsen <>
Subject: Re: Last Call: HTTP Extension Framework to Proposed Standard

First, I am sorry for the delay - my mailer chose to crash instead of
sending the message so I had to reconstruct it the hard way :(

>I have just finished looking at draft-frystyk-http-extensions-02.  Not all
>of my problems with the 01 version got resolved in this draft.  After some
>careful reading I found that protocol elements which confused me when
>reading the 01 draft are actually just plain broken.  I have to reverse my
>earlier conclusion that the protocol is 'reasonably sound': in fact
>draft-frystyk-http-extensions-02 has serious protocol flaws which make it
>unfit to go to proposed. 

Ok, let's go through your comments.

>Overall it looks to me like the Ext: header part of the protocol and
>the cachability parts need more work.  The draft is also very short on
>advice to implementers and does not have a high-level overview of
>protocol interactions before going into specifics.

I can't comment on the first half of this paragraph but regarding the
editorial composition of the document, you are the only one wanting a
longer document, others expressed that they were happy as it is [1]. In
fact, several comments were attempts to make further simplifications [2].

>By allowing for cachable extended responses, the spec opens up a
>rather large can of worms.  Below I have identified several places
>where the spec does not handle these worms with sufficient care, but
>my list is probably not complete.  In my opinion this stuff should be
>fixed by avoiding the problem all-together, i.e. by simply making all
>extended responses uncachable, and making sure that extended requests
>can never be responded to by a plain HTTP/1.x cache.

That would certainly be one solution, however, I think it would be
unfortunate to propose an extension mechanism to a protocol that short
circuits the several years of detailed caching design that has been going
on in HTTP/1.1.

>A detailed list of problems follows.
>- Sec 4 last para:
>       Optional
>       declarations MAY be applied to any HTTP message without any change to
>       existing HTTP semantics
>This seems to say 'if you apply the declaration to any message, the
>HTTP semantics will not change'.  However if extension headers are
>applied to a HTTP/1.1 304 message, the message becomes an incorrect
>1.1 message which (worse) can cause all kinds of consistency problems
>in caches.  Some kind of warning that one cannot piggyback extensions
>onto 304 messages is in order, I find the spec quite misleading to
>implementers as far as HTTP/1.1 pitfalls are concerned.

The HTTP extension framework doesn't break HTTp/1.1 (not any other version
of HTTP). That is, whatever is illegal in HTTP/1.1 is also illegal when
using HTTP extension declarations. Your 304 example is one such situation,
another would be to not honor the connection header field. I think listing
them all would be a mistake as this is essentially what HTTP/1.1 does already.

However, I think you bring up an interesting point and that is that the
sentence "without any change to existing HTTP semantics" is not clear. What
is meant is that the recipient is free to ignore the optional extension
declaration without affecting the outcome of handling the message. However,
this is stated just above (see section 4.0):

	An optional extension declaration indicates that the ultimate
	recipient of the extension MAY consult and adhere to the rules
	given by the extension when processing the message, or ignore
	the extension declaration completely. 

and also in section 5.1, numbered item 1):

	1. Identify all mandatory extension declarations (both
	hop-by-hop and end-to-end); the server MAY ignore optional
	declarations without affecting the result of processing the
	HTTP message;

I therefore suggest shortening the above sentence to say

	Optional declarations MAY be applied to any HTTP message if

>- Sec 5 point 3:
>          3.  If 2) did not result in a 510 (Not Extended) status code, then
>              strip the "M-" prefix from the method name and process the
>              remainder of the request according to the semantics of the
>              extensions and of the existing HTTP method name as defined in
>              HTTP/1.1 [5] or later versions of HTTP.
>This implies that a proxy which has processed an end-to-end mandatory
>extension should _not_ strip the existing Man header in the request
>when forwarding the request towards the origin server (when
>'processing the remainder according to the semantics of the existing
>HTTP method name as defined in HTTP/1.1').  Not stripping the header
>is of course wrong.  If the proxy adds its own M- extension to the
>outgoing request without stripping the processed Man header first,
>this will cause the ultimate recipient of that extension to process
>the old Man extension again.  This will have fun effects especially if
>the extension is for a payment protocol.

The algorithm that you refer to says nothing about how to *forward* a
mandatory request but how to *respond* to a mandatory request:

	An ultimate recipient conforming to this specification receiving
	a mandatory request MUST process the request by performing the
	following actions in the order listed below:

In the next paragraph, it is stated how to *forward* a mandatory request in
case a proxy is *not* the ultimate recipient:

	A proxy that does not act as the ultimate recipient of a
	mandatory extension declaration MUST NOT remove the extension
	declaration or the "M-" method name prefix when forwarding
	the message.

In order for a server (any server be it a proxy or origin server) to
fulfill a mandatory request, it MUST understand ALL mandatory extension
declarations in that request. If it doesn't, it can either return an error
or tunnel the request in case it is not the origin server. It is not
possible to pick a mandatory piece of the request and not another mandatory
piece and return OK.

How to forward the request is defined by HTTP and this doesn't change in
this specification. Therefore, I think the text is exactly correct in that
it doesn't make the assumption you infer, namely that the only right way is
to strip the extension.

>In the table of section 14 it says 'may strip', which conflicts with
>the language above, and is still wrong because it must be MUST strip.
>- Sec 5 point 4:
>              A server MUST NOT fulfill a request without
>              understanding and obeying all mandatory extension
>              declaration(s) in a request.
>This implies that a request cannot contain two mandatory extensions
>which are to be executed by different upstream servers.  For example,
>if I want my browser to send a mandatory extension to my firewall
>proxy to enable some privacy processing, I cannot at the same time use
>a browser plugin which sends a mandatory extension to trigger some
>plugin-related functionality in the origin server.  Not being able to
>combine two extensions in this way is a rather severe restriction
>which kills the whole protocol for me.  It means that plugin vendors
>cannot use the protocol for fear of proxies getting in the way, and
>that proxy vendors cannot use the protocol for fear of plugin vendors
>getting in the way.  Result: nobody can use it.

This has in fact little to do with the extension framework. What you are
questioning is the basic notion of scope in HTTP/1.1. HTTP/1.1 has two
scopes: hop-by-hop and anything else (which we often call "end-to-end"
although it really depends on how you look at it). What I think you are
trying to do is to introduce a 3rd scope meaning some intermediary proxy
down the line. HTTP extensions do not attempt to change/extend the scope of
HTTP and hence doesn't has anything to say about a 3rd scope. This is
stated in section 4.1 and 4.2 respectively:

	End-to-end declarations MUST be transmitted to the ultimate
	recipient of the declaration. 
	Hop-by-hop extension declarations are meaningful only for a
	single HTTP connection

>The Ext: mechanism should be fixed to solve this problem, e.g. the
>Ext: header should include the names of the extensions that were
>understood and obeyed by different upstream servers.
>- Sec 5 para after enumerated list:
>       HTTP proxies that
>       do not understand the "M-" method name prefix SHOULD return 501 (Not
>       Implemented) or turn themselves into a tunnel (see [5]) in which case
>       they do not take any part in the communication.
>This seems to be a re-statement of a requirement that the HTTP/1.1
>spec [5] makes on plain HTTP/1.1 proxies which do not implement
>mandatory.  However, as far as I know, this is a particular
>interpretation of [5] that is wrong: [5] does not require proxies to
>turn themselves into a tunnel, they can forward the request and cache
>the response.  The revised 1.1 spec (draft-ietf-http-v11-spec-rev-06)
>is a bit more clear on this.

An HTTP cache can not serve a cached document in response to a method it
doesn't understand. This was discussed on the HTTP Working Group mailing
list [4] where the proposed change didn't go into the rev-06 (I should know
as I was the one objecting - exactly for this reason).

>- Sec 5.1 Fulfilling a Mandatory Request
>There is no equivalent material on how to fulfill an optional request.

It is stated in section 4:

	An optional extension declaration indicates that the ultimate
	recipient of the extension MAY consult and adhere to the rules
	given by the extension when processing the message, or ignore
	the extension declaration completely

I don't know what else to say - if you feel this isn't adequate then please
let me know.

>- Sec 5.1
>       In order to avoid that the Ext header
>       field inadvertently is cached in an HTTP/1.1 cache, the response MUST
>       contain a no-cache cache-control directive.
>This caching discipline is wrong for some extensions, for example
>those which `introduce new encodings' (see introduction).  For
>example, for an extension which has the same effect as the HTTP/1.1
>accept-encoding header, the appropriate caching discipline is
> request:
>  M-GET /p/q HTTP/1.1
>  Man: "http://www.x.y/accept-encoding"; ns=16-
>  16-accept-encoding: gzip, identity;q=0
>  ....
> response:
>  HTTP/1.1 200 OK
>  Ext:
>  Vary: man, 16-accept-encoding
>  Date: Sun, 25 Oct 1998 08:12:31 GMT
>  Expires: Sun, 25 Oct 1998 08:12:31 GMT
>  Cache-Control: max-age=100000
>  16-content-encoding: gzip
>  ....
>  [gzip-encoded entity body]

I think you are misunderstanding a very fundamental piece of the HTTP
extension framework: the scope of an extension declaration is the current
*message* and not the current *request/response pair*. In your example, the
header field in the response:

	16-content-encoding: gzip

is not defined by this specification - it is undefined. If the server has
applied a new content-encoding to the response, it must say so by adding an
extension declaration:

	HTTP/1.1 200 OK
	Vary: man, 16-accept-encoding
	Date: Sun, 25 Oct 1998 08:12:31 GMT
	Expires: Sun, 25 Oct 1998 08:12:31 GMT
	Cache-Control: max-age=100000, no-cache="Ext"
	Man: "http://www.x.y/accept-encoding"; ns=16
	16-content-encoding: gzip

In this case, the Ext header field is really only an "ACK" that the server
handled the extension in the request and didn't just try to fool you. It
should therefore not be cached.

The scope of extension declarations is described in section 3

	An extension declaration can be used to indicate that an
	extension has been applied to a message and possibly to
	reserve a part of the header namespace identified by a header
	field prefix (see 3.1). 

and later in the same section

	This specification does not define any ramifications of
	applying an extension to a message nor whether two extensions
	can or cannot logically coexist within the same message. 

and explicitly for header-prefixes in section 3.1:

	All header fields in the message that match this string,
	using string prefix-matching, belong to that extension
	declaration. Header field prefixes allow an extension
	declaration to dynamically reserve a subspace of the header
	space in a protocol message in order to prevent header field
	name clashes and to allow multiple declarations using the
	same extension to be applied to the same message without

Nowhere is the term *request/response pair* (or equivalent) used. However,
the misunderstanding is important in itself and so I have added a sentence
to the  2nd last paragraph of section 3:

	Note, that consistency is no substitute for including an
	extension declaration in the message: header fields with
	header-prefix values not defined by an extension declaration
	in the same message are not defined by this specification.

>There should be no 'no-cache="ext"', because the extended response
>remains valid, the extension remains applied, for future matching
>requests.  In fact if we have 'no-cache="ext"', then a cache will
>respond with the following if the above request is repeated:
>  HTTP/1.1 200 OK
>  Vary: man, 16-accept-encoding
>  Date: Sun, 25 Oct 1998 08:12:31 GMT
>  Expires: Sun, 25 Oct 1998 08:12:31 GMT
>  Cache-Control: max-age=100000
>  16-content-encoding: gzip
>  ....
>  [gzip-encoded entity body]
>And this will cause the client to conclude that it is dealing with a
>'broken server [giving] the false impression that an extended request
>was fulfilled' (first para of 5.1. At least I assume this is what the
>client is supposed to conclude when Ext is missing. At is nowhere
>spelled out explicitly that clients should check for ext: at all!)

You are misunderstanding HTTP here - an HTTP cache can not serve a document
in response to a request with a method it doesn't understand. Imagine that
you have a new method called "ORDER-PIZZA". A response to this request may
be a pizza and a big cachable image but unless you understand the method,
you can't serve that image from the cache. Otherwise, the user could end up
with the image and not a pizza!

Again, this was discussed in [4] on the HTTP Working Group mailing list.

>- Sec 5.1
>       If the
>       response is otherwise cachable by HTTP/1.1 caches, the server SHOULD
>       include an appropriate max-age cache-control directive:
>The unqualified SHOULD here is wrong.  First, servers which are not
>the origin server often have not enough information to invent an
>`appropriate' max-age value.  Also, adding a max-age can weaken the
>cachability of the response in some cases (e.g. when it already had an
>expires: <now>, which will lead to problems.

The max-age cache-control directive always takes precedence over the (old)
Expires header field so this wouldn't cause any conflicts. In fact, the
examples in the document use both in order to allow HTTP/1.1 to do more
advanced caching than HTTP/1.0 caches.

That being said, I think that MAY is sufficient in this contest, especially
as you say, it is difficult to dream up an appropriate value.

>- Sec 7
>       The server SHOULD send back all the information necessary for the
>       client to issue an extended request. It is outside the scope of this
>       specification to specify how the extensions inform the client.
>If the spec does not specify how to fulfill a SHOULD, then no client
>can be unconditionally compliant until some other spec specifies this.
>This is a bit of a problem; the SHOULD should be a MAY.

No, it is not a MAY - in fact it is a lowercase should.

>- Sec 9
>       If an extension does
>       change the default cachability of the response, the originator MUST
>       include a Cache-Control header field containing the cache directives
>       corresponding to the desired result of the extended semantics.
>The above MUST hands the can of worms I talked about earlier to most
>extension implementers.  For a start, just _including_ a Cache-Control
>is not sufficient.  A Cache-Control or Expires header which is already
>present in the response could weaken the directives in the newly
>included Cache-Control, leading to inappropriate caching.  This works
>both ways: an included Cache-Control could also weaken already
>existing directives.  So it is not that simple, one has to somehow
>merge the directives in the old Cache-control and Expires with the new
>directives, to produce a set of directives which is stronger than
>both.  This can be done, though HTTP/1.1 was not really designed to
>support it.  So it will be a _very_ hairy operation, especially if
>extension-directives are present.  One could of course take the
>obvious way out and put in 'Cache-control: no-store' which is the
>strongest directive possible.

First, your comments are more intended for the complexity of HTTP/1.1 than
of the extension framework: yes, cache-control and vary header fields can
be hard to deal with.

Second, you are assuming that the new combined cache-control must be
stronger than what it replaces. This doesn't have to be the case.

Third, the word "include" refers to what goes into the forwarded message.
That is, it has already been merged with whatever was already there in the
message being forwarded.

Forth, I don't understand why you say that "no-store" in the obvious
solution. There is no obvious solution as it depends on the extension and
how it is being used.

Fifth, where does it say that HTTP/1.1 wasn't designed to deal with this?

>Finally, if an extension does changes the default cachability of the
>response, it may also be necessary to reflect this by adding/updating
>Vary, Expires, and Warning headers.

Good point - we should add a sentence to that effect:

	The originator MAY also use the Vary and the Expires header
	fields if appropriate.

>- Sec 15.1, Vary example
>This example was apparently adapted from an e-mail message I wrote.
>In that message I noted that correct use of the vary header was
>slightly non-trivial.  I intended the 'transform' extension to have a
>different effect on the response entity depending on which
>transformation was used so my vary header was
>  Vary: man, 16-use-transform
>but the vary header I see in the draft is
>  Vary: man
>which is *incorrect* for the extension semantics I had in mind.  The
>text next to the example says 'varies on the request extension
>declaration' but what I had in mind was 'varies on the request
>extension declaration and its associated parameter in its header

I can't comment on that the example is different from what you expected and
that this makes it incorrect in your eyes. It is not incorrect. I didn't
use your example (although correct) because I don't want to give the
impression that header-prefix'ed header fields are defined without an
extension declaration - they are not.

>- Sec 15
>I have said before (in reviewing the 01 draft) that I am not that
>happy with the caching discipline shown in the examples, but I won't
>repeat that here because I have already said enough about caching.

Thanks for your comments - I hope my answers clarify your questions. I will
leave it up to the IESG to determine whether your points amount to "serious
protocol flaws" and "plain broken" (or should I say high crimes and


Henrik Frystyk Nielsen,
World Wide Web Consortium