- From: Rohit Khare <khare@pest.w3.org>
- Date: Mon, 19 Aug 96 16:37:13 -0400
- To: internet-drafts@ietf.org
- Cc: jepi-core@commerce.net, http-wg%cuckoo.hpl.hp.com@hplb.hpl.hp.com
HTTP Working Group Rohit Khare
INTERNET-DRAFT World Wide Web Consortium
<draft-ietf-http-pep-02> Massachusetts Institute of Technology
Expires: February 19, 1997 August 19, 1996
HTTP/1.2 EXTENSION PROTOCOL (PEP)
Status of this memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress".
To learn the current status of any Internet-Draft, please check the
"1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
ftp.isi.edu (US West Coast).
Distribution of this document is unlimited. Please send comments to
the HTTP working group at <http-wg@cuckoo.hpl.hp.com>. Discussions of
the working group are archived at
http://www.ics.uci.edu/pub/ietf/http/.
This Internet Draft is also available as W3 Consortium Working Draft
WD-http-pep-960819. A current version is available at
http://www.w3.org/pub/WWW/TR/WD-http-pep.
Abstract
PEP is a system for HTTP clients, servers, and proxies to reliably
reason about custom extensions to HTTP. Traditionally, HTTP agents
offer extended behavior by private agreement using additional message
headers. PEP has features for expressing the scope, strength, and
ordering of such extensions, as well as which extensions are
available.
PEP as described here is substantially simpler than previous drafts.
The title has also been changed, since PEP addresses a chartered HTTP
WG work item.
1. Introduction
HTTP [3] messages, like most applications of RFC 822 [4], can be
extended with additional header fields and content formats.
Traditionally, the standards update process itself extends the
protocol for major, global new features. However, this approach is
problematic for deploying casual experiments which could scale up from
pairwise to universal deployment, especially as HTTP agents become
more modular, dynamic, and diverse. Casual extension of HTTP is
hindered by several challlenges:
1. Naming. To guarantee interoperability, header names, encoding
tokens, types, response codes, etc. must be centrally arbitrated
by appealing to IETF standards-track documents or the Internet
Assigned Numbers Authority (IANA). This requires more effort from
implementors than an extension may warrant.
2. Importance. There is no way to indicate the consequences of
ignoring an extension: can an agent silently ignore it, may it
warn the user, or must it return an error message?
3. Scoping. There is no reliable way to indicate who is to process an
extension: the next HTTP agent (proxy), or only the origin agent
(client or server)? 1.1's Connection: is only a partial solution,
since it cannot convey the criticality of stripped headers.
4. Ordering. There is no way to indicate in what order to process
extensions. If several are applied, it can be critical to 'unpack'
the message in the correct order.
5. Initiation. Short of unilaterally utilizing an extension, there is
no way to request the other party to initiate it.
6. Advertising. Though it is possible to enumerate content variants
and the headers they differ upon, it is not possible to describe
available extensions in detail.
7. Inquiring. There is no way to inquire if an extension is supported
before attempting to use it. This is critical for selecting among
related extensions and upgrading to new versions.
PEP introduces the concept of protocol extensions to systematically
address these issues at a level of abstraction above header names and
content-types (hence, `Protocol Extension Protocol'). With PEP, HTTP
agents can interoperate correctly with known and unknown protocol
extensions, select protocol extensions available to both sides, and
query partners for specific capabilities. PEP adds to HTTP the
extensibility lessons learned from ESMTP (extension naming) [10], IPv6
(unknown-option disposition) [5], and Telnet (option negotiation)
[12].
In addition to reliably describing statically extended HTTP servers
and clients, PEP will work with dynamically extended agents. Indeed,
the authors expect that PEP will drive the deployment of a new
generation of exensible agents (such as W3C's Jigsaw server and libWWW
reference library).
NOTE: PEP has been developed over the last year by many interested
parties, including the W3C/CommerceNet Joint Electronic Payments
Initiative (JEPI). Two strongly related drafts are also available from
the IETF and W3C: Don Eastlake's Universal Payment Preamble [6] and
``Selecting Payment Mechanisms Over HTTP (or, seven examples of UPP
over PEP)'' [9].
1.1 EXAMPLE: PICS
Consider the example presented by the first deployed PEP application,
the Platform for Internet Content Selection (PICS) [11]. PICS
describes a format for labeling almost any content. Since it's
intended to be a fairly ubiquitous addition to various Internet
protocols, PICS appropriates the PICS-Label: header for 822
applications, including mail, news, and web content. The spec could
not use, say, Label:, since there's no way to effectively arbitrate
between conflicting uses of header names (PICS was not developed under
the IETF standards track).
Even with an eventual dispensation from IANA to use this new header,
new issues arise: If a server attaches this new header to label an
HTTP response, will a caching server pass it along? Will it act upon
the label at all? What kind of feedback will the end-user get when
this ``unknown'' header appears? How can the user specify which rating
systems to report?
The direct fix to the first few problems is to add a
machine-understandable statement about the extension and its new
headers. Protocol: headers describe the name of the extension, the
scope, importance, and any associated data headers. Moving further,
though, additional information about the protocol can be added to that
directive: which PICS services were employed, etc.
Using PEP in this way also solves the next logical problem: how can a
client request a server to initiate vending PICS labels? The same
structure for a Protocol: can be reused in a Protocol-Request: asking
the counterparty to start using the specified extension.
In addition, PEP affords additional capabilities, like using
Protocol-Query: to test if PICS is available and Protocol-Info: to
advertise it. In all cases, the PICS protocol extension can specify
additional parameters, such as which rating systems, services, which
kinds of labels, and which other URLs are labelled.
2. The PEP Model
Protocol extensions can modify HTTP messages, most commonly by adding
headers, but also by modifying the contents, much like a proxy server
might do today. There are four points at which PEP processing occurs
in a basic HTTP transaction (see Section 3 for usage descriptions at
each step):
1. Sending an HTTP Request
2. Receiving an HTTP Request
3. Sending an HTTP Reply
4. Receiving an HTTP Reply
During a conversation, either side may invoke a Protocol:, issue a
Protocol-Request:, advertise some Protocol-Info:, or test a
Protocol-Query:. PEP-compliant recipients decode extended messages and
reply to each Protocol-Request: and Protocol-Query:.
Unlike some previous extension proposals [9], and unlike some existing
HTTP/1.1 features (Upgrade: and Connection: tokens), extensions are
attributes of a resource rather than a server. This means all PEP
statements are applied to the single resource mentioned in any HTTP
message (though there is a facility to hint at which other resources
the same statement applies to).
An HTTP agent can begin using a protocol extension (or query for one)
because:
* It is configured to do so by a user or webmaster
* It has been requested to do so by the other party
* The other party has hinted that an extension is available
* Someone else has hinted that an extension is available
Another aspect of the PEP model is that protocol extensions can have>
negotiable parameters. Unlike previous proposals which model
extensions as binary have/don't have features, PEP expects protocol
extensions will be able to handle different versions of a particular
protocol, invocation of ``compatible protocols'', and the selection of
compatible modes from several that might be advertised.
To summarize, PEP as proposed here will allow HTTP/1.2 implementors to
make the following kinds of statements:
* ``This agent has and is using (or does not have and isn't using)
the http://w3.org/DES protocol extension with mode parameter CFB
and header DES-Info:''
* ``This agent requires/requests/forbids the recipient to begin
using the http://w3.org/DES protocol extension with mode parameter
ECB''
* ``This agent has/requires/refuses the http://pics.org/PICS-1.1
protocol extension available for any URIs beginning with
http://www.my.site/homepages/''
* ``Does the recipient have the http://pics.org/PICS protocol
extension available for service parameter http://rsac.org/rsaci
?''
* ``This agent has applied two protocol extensions to this message
in this order.''
* ``This agent is using the `generic' http://w3.org/UPP payment
negotiation extension via the payment protocol
http://cybercash.com/CCCP.''
3. PEP Usage
Before presenting specifics about each of the four PEP directives and
the four points where they are processed, here are a few common
points:
* Attribute-value bags are used to represent all the data carried in
PEP headers.
* Each directive specifies which agent must process it and the
consequences of not processing it.
* Each directive is associated with a set of resources: either the
resource represented by the HTTP request or reply, or a list of
resource and resource prefixes it applies to.
A bag is an attribute-value list, where bags are allowed recursively
as values as well. In PEP, each directive begins with values
describing the protocol (making the extension name the top attribute).
Some of those values are common to all directives: the strength (str),
scope (scope), for list (for) , extension-specific parameters
(params), and other headers associated with this extension (headers).
The formal BNF is included in Section 3.5.1.
Strength and scope are important common concepts. Any directive can be
ignored, but strength specifies whether an error should be generated
as a result. Similarly, scope defines which agents should handle the
directive at all; either the immediate (conn) or final (origin)
recipient. The combination of the two concepts makes PEP powerful
enough to deploy features that in the past required a minor-version
number bump. With PEP, for example, Content-Transfer-Encoding: could
be described as {str req} {scope conn} for rapid experimentation with
compressed headers, sticky headers, etc.
A word about the for list: Since each PEP directive applies only to
the resource at hand (rather than, say, the entire server or all
resources of a given type), a hint as to which other resources the
directive applies to will prevent negotiation round-trips each time.
Thus, many PEP applications could include {for /*} to apply to an
entire server. Remember, there already are examples in HTTP/1.1 of
features which only apply to particular resources, such as Range:.
When fetching a URL, a client will typically search through its
database of hints to find all applicable directives before sending an
HTTP request. Of course, an agent which does not want to use hints can
ignore them: if the for list includes the URI of the current message,
just throw away the list; if it does not, just throw away the entire
directive.
Finally, note that the Protocol-Query: / Protocol-Info: pair describes
hypothetical capabilities of agents and resources but that the
Protocol-Request: / Protocol: pair has immediate consequences for the
current HTTP transaction. Each pair hangs together and must be
implemented together, though only the latter pair is mandatory.
3.1 PROTOCOL-QUERY
The first statement in a conversation might be to ask if the
counterparty supports an extension at all. The Protocol-Query: header
lets one agent ask the other if a particular extension is supported;
the response comes back in a subsequent Protocol-Info:.
A query directive specifies its scope, but not a strength, which
defaults to opt; that's because a query always asks optionally for
more information, but can neither demand an answer (since query
support is optional), nor refuse one.
An agent can query if an extension is supported for an entire list of
URIs. This means that an answer is expected when the replying agent
sends a message about any resource on the list; and that an answer
should reveal as much as can be said about extension support for that
URI set. This way, a server can ask for a client to be prepared to use
an extension in a part of his site (e.g. UPP in /Catalog/*); or a
client can ask a server if an extension applies to all its documents
of the same type (e.g. indexed range retreival for *.pdf).
3.1.1 Sending an HTTP Request
If a client believes an extension may be supported, it can attach a
Protocol-Query: to an outgoing request to test if an extension is
supported for that URI (empty for list), an entire server ({for /*})
or something in between.
A client can wait to piggyback its queries on other HTTP transactions
or it can explictly issue an OPTIONS request with its queries.
Note that a protocol extension might be defined to have side-effects
when it is queried. For example, the UPP payment selection protocol
defines that the answer to any queries for UPP should include
additional information about particular payment mechanisms. Thus, in a
dynamically extended agent an agent should call back the extension to
handle a query.
3.1.2 Receiving an HTTP Request
When an agent receives a query, it should respond whether the stated
configuration is acceptable or refused for any of the URI(s)
specified. Upon receipt of a query, an agent should test if the
extension is available and, if so, its response to the proposition.
The server should be prepared to send the resulting information in its
reply.
3.1.3 Sending an HTTP Reply
In preparing an HTTP reply, a server must respond to queries in the
HTTP request, as well as issue its own queries on the reply. For the
responses, see Section 3.2 below.
3.1.4 Receiving an HTTP Reply
The answers returned to the client are in the form of Protocol-Info:;
see Section 3.2. The server may have attached queries of its own; the
client should prepare an answer and attach it to the next HTTP request
it makes for a URI which matches the realm of the query. Here, too, a
dynamically extended client should immediately call back an extension,
since an extension's response to Protocol-Query: may result in
immediate side effects (e.g. a payment selection interface appearing
with highlighted possible selections).
3.2 PROTOCOL-INFO
The second statement in a conversation might be to declare that one
party requires/allows/forbids a given protocol extension, either
unilaterally or in response to an earlier query. The Protocol-Info:
header lets one agent tell the other if a particular extension is
supported; there is no required response. Remember, though, that
Protocol-Info: itself is the required response to Protocol-Query: (see
Section 3.1).
An informational response states if the protocol is required, allowed,
or forbidden for all matching URIs.
3.2.1 Sending an HTTP Request
A client will add Protocol-Info: directives if required by a previous
server query (Section 3.1.4) or if it has any unilateral declarations
to advertise.
3.2.2 Receiving an HTTP Request
A server should take note of any available information; knowledge of a
client's capabilities is important for step #3 (Section 3.3.3 and
3.4.3).
3.2.3 Sending an HTTP Reply
A server will add Protocol-Info: directives if required by a previous
client query (Section 3.1.1) or if it has any unilateral declarations
to make.
3.2.4 Receiving an HTTP Reply
A server should take note of any available information; knowledge of a
client's capabilities is important for step #1 (Section 3.3.1 and
3.4.1). The response can be to either ignore this information, request
initiation, or directly initiate the extension on the next request.
3.3 PROTOCOL-REQUEST
The third statement in a conversation might be to directly demand or
forbid that the counterparty begin using an extension.
Protocol-Request: lets one agent tell the other to initiate a
particular extension; the response requires a subsequent Protocol:
directive for each request.
3.3.1 Sending an HTTP Request
When a client wants to tell the server to begin using some extension,
it should check if it knows that the extension will work because of
stored Protocol-Info:. The Protocol-Request: describes the desired
protocol; it may include a range of parameters for the respondent to
pick and choose from.
3.3.2 Receiving an HTTP Request
A server must test if a request will succeed. If so, the server must
begin using the specified extension. If it cannot, the preferred
mechanism is to echo back the same Protocol-Request: but with {str
ref}. In addition, if the original request was {str req}, then
indicate that the request failed with a 42x client or 52x server error
status code (Section 3.5.2).
3.3.3 Sending an HTTP Reply
A server must reply to the client's request to require, accept or
forbid an extension by either initiating or preventing use of the
specified protocol or a compatible equivalent (Section 3.4.3). At this
point, a server may also attach its own requests for a client to
initiate extensions when accessing this URI or others.
Note that if a server Protocol-Request: later fails to elicit a
Protocol: response from the client, nothing can be assumed about the
client's ability to do PEP, since it may have ``forgotten'' the
request in the meantime.
3.3.4 Receiving an HTTP Reply
The client must evaluate each server Protocol-Request: and, if valid,
be prepared to use the specified protocol extension when accessing any
URI matching the request's for list (Section 3.4.1). If it is not
willing or able to use the specified protocol, it must echo back an
error on the next access as in Section 3.3.2.
Note that if a client Protocol-Request: fails to elicit a Protocol: or
PEP error response (assuming the response URI is in the request's
realm) at, the client must conclude the origin server is not
PEP-compliant.
3.4 PROTOCOL
The final statement in a conversation might be to use an extension
directly. Protocol: lets one agent declare that the particular
extension is in use for the current message; the response may be to
continue using the extension. An agent can apply an extension at any
time; the preceeding three conversational steps of testing and
requesting are only an example.
This directive packages together any added header lines and any
reprocessing of the message body. Since HTTP/1.1 eliminates many
earlier ambiguities about the ordering of headers, we can state
categorically that protocol directives must be dispatched in the order
they are listed in the Protocol: header. [The Content-Encoding:
pipelining mechanism of earlier proposals has been scrapped].
It is worth noting that the application of a protocol extension can
affect the cacheablity of the response. This document does not define
a cache control directive, but expects it to be feasible within the
1.1 model.
Connection-scope extensions can remain backward-compatible by listing
every associated header and the name of the extension in the
Connection: header.
3.4.1 Sending an HTTP Request
The client must decide which extensions to apply to the current
request: it must respond to each applicable Protocol-Request:, it can
continue using extensions the server has already used for the
request-URI, and it may initiate others. As each extension is applied,
added headers are referenced from the Protocol: line in order. If some
Protocol-Request: is satisfied by some other protocol extension
compatible with the one requested, the client will issue two protocol
directives, one for the extension used, and another indicating the
extension requested is being used via the other.
3.4.2 Receiving an HTTP Request
The server evaluates each protocol extension which is in scope
according to the order it is listed in the Protocol: header.
As the server evaluates each protocol, it can interrupt and return
HTTP error status codes: if a client encoding error is detected (42x),
the server can't or won't decode properly (52x), or if a protocol is
used inappropriately (it's refused, not allowed in conjunction with
certain other protocols, missing, etc.).
Proxy servers operate similarly: they can simply remove unknown
optional connection-scope directives but must report errors for
required directives.
3.4.3 Sending an HTTP Reply
The server must also choose protocols it will use from the sets of
those requested by the client, used by the client, and those it will
initiate independently. If it cannot construct a valid reply due to
contradictions among those requirements, it must return HTTP error
status 520 and enumerate the server's constraints in Protocol-Request:
headers of the error message.
3.4.4 Receiving an HTTP Reply
The client evaluates each protocol extension which is in scope
according to the order it is listed in the Protocol: header.
As the client evaluates each protocol, it could fail for many reasons.
One recovery technique is to reissue its request with the offending
protocol refused in a Protocol-Request: header. Errors from executing
required extensions should be reported to a user or log.
3.5 SYNTAX
PEP-related syntax is specified here relative to the definitions and
rules of the HTTP/1.0, HTTP/1.1, and the relative URL specification.
3.5.1 PEP Headers
PEP defines two new required general header fields, Protocol: and
Protocol-Request:, as well as two new optional general header fields,
Protocol-Query: and Protocol-Info:.
/* Added to General Header rule, Sec 4.3 of HTTP/1.1 */
Protocol = "Protocol" ":" 1#bag
Protocol-Request = "Protocol-Request" ":" 1#bag
Protocol-Query = "Protocol-Query" ":" 1#bag
Protocol-Info = "Protocol-Info" ":" 1#bag
/* Following rules are copied from Section 2.2 of HTTP/1.1 */
bag = "{" bagname 1*LWS *bagitem "}"
bagname = token | URI
bagitem = bag | token | quoted-string
word = token | quoted-string
token = 1*<any CHAR except CTLs or tspecials>
tspecials = "(" | ")" | "<" | ">" | "@"
| "," | ";" | ":" | "\" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT
quoted-string = ( <"> *(qdtext) <"> )
qdtext = <any CHAR except <"> and CTLs but including LWS>
Each bag has several top-level reserved attributes defined:
{<protocol-identifier> {scope (origin | conn)}
{str (opt | req | ref)}
{headers *<token>}
{params ...}
{for *<uri>}
{via <protocol-identifier>}}
protocol-identifier
The well-known protocol extension URI or an IANA-registered
token.
scope
scope is either of "conn" or "origin" ; the default is
"origin". It defines which agent must recognize this directive:
the immediate recipient or the origin client/server.
str
strength is one of "req", "ref" or "opt" ; the default is
"opt". In particular, a protocol directive only allows req or
opt; request and informational directives allow all three; and
queries only allow opt.
headers
A list of associated message headers; the default is the empty
set. If a header token ends in "*", it must be used to refer to
all message headers matching that prefix. Each directive
enumerates its own associated headers to disambiguate
`ownership'. This information is critical when 1) reporting an
error, 2) stripping an extension off of a message (if
possible), and 3) allowing multiple applications of an
extension.
params
A list of parameters configuring the extension, according to
the protocol specification; the default is empty.
for
a list of relative or absolute URIs [1,2,7] which a protocol
request is binding upon; the default when the for list is
empty, is the request-URI or Location: response header. If a
reference ends in "*", it should be understood as a wild-card
character (particular implementations may choose to only
recognize restricted applications of "*", e.g. `only in the
last component of http: URLs').
via
When responding to a request addressed to one protocol
extension (a query or request directive), an agent may answer
via another (more specific, upgraded, compatible) extension
(token or URI). The latter extension is expected to be used
elsewhere in the same message. This is a hint and only allowed
in the Protocol: and Protocol-Info: directives. The default is
empty.
Note: Multiple occurrences of PEP-defined attribute names in a list
("{str req}...{str ref}") will yield undefined behavior.
3.5.2 PEP Status Codes
PEP defines several new status codes for HTTP replies. Note that the
HTTP/1.1 specification states in Section 6.1.1:
The first digit of the Status-Code defines the class of response.
The last two digits do not have any categorization role.
Informally, PEP distinguishes x2z response codes:
200 Class
220 Uses Protocol Extensions
400 Class
420 Bad Protocol Extension Request
421 Protocol Extension Unknown
422 Protocol Extension Refused
423 Bad Protocol Extension Parameters
500 Class
520 Protocol Extension Error
521 Protocol Extension Not Implemented
522 Protocol Extension Parameters Not Acceptable
Each of 400 and 500 class responses may include entity bodies with an
explanation of the error, and an indication of whether the problem is
temporary or permanent. A 220 response should be used only when some
PEP feature is required for correct handling -- it affects the
cacheability of the response.
4. Conclusions
This is a discussion draft, not a complete specification. In the
meantime, this will be the basis for all further JEPI work.
For additional details and explorations of various issues (e.g.
security concerns), readers are encouraged to study the earlier
drafts, available at http://www.w3.org/pub/WWW/TR/WD-http-pep.
5. References
[1] Berners-Lee, T., "Universal Resource Identifiers in WWW: A
Unifying Syntax for the Expression of Names and Addresses of
Objects on the Network as used in the World-Wide Web", RFC
1630, CERN, June 1994.
[2] Berners-Lee, T., Masinter, L., and M. McCahill, Editors,
"Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox
Corporation, University of Minnesota, December 1994.
[3] Jim Gettys, et al, "Hypertext Transfer Protocol -- HTTP/1.1".
Internet Draft W3 Consortium/MIT, August 1996 (Work in
Progress).
[4] D. H. Crocker. "Standard for the Format of ARPA Internet Text
Messages." STD 11, RFC 822, UDEL, August 1982.
[5] S. Deering, R. Hinden, Editors, "Internet Protocol, Version 6
(IPv6) Specification", RFC 1883, December 1995.
[6] D. Eastlake, "Universal Payment Preamble", Internet Draft
CyberCash, March 1996 (Work in Progress).
[7] R. Fielding. "Relative Uniform Resource Locators." RFC 1808 ,
UC Irvine, June 1995.
[8] JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft,
August 1996 (Work in Progress). [Also available as
http://www.w3.org/pub/WWW/Payments/JEPI/draft-jepi-uppflow-00
.html]
[9] D. Kristol, "A Proposed Extension Mechanism for HTTP", Internet
Draft, January 1995 (Work in Progress, Expired).
[10] J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker.
"SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach
Consulting, Network Management Associates, Brandenburg
Consulting, November 1995.
[11] J. Miller. "Label Syntax and Communication Protocols (Version
1.1)." W3C Proposed Recommendation PR-PICS-services , W3
Consortium/MIT, August 1996.
[12] J. Postel, J. Reynolds, "Telnet Protocol specification." STD 8,
RFC 854, USC/ISI, May 1983
6. Author's Address
[Note: a great many people have participated heavily in the research,
design, development, and description of the ideas presented here.
Future revisions will sort out the contributions of various folks,
likely adding co-authors &c. In the meantime, I will be the single
contact point for this draft. Many, many thanks to these folks -- you
know who you are!]
Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org
Web: http://www.w3.org/People/Khare
Received on Monday, 19 August 1996 13:37:45 UTC