W3C home > Mailing lists > Public > ietf-discuss@w3.org > March 1999

[Seventh Heaven] Need help describing APPLCORE

From: Rohit Khare <rohit@fdr.ics.uci.edu>
Date: Mon, 08 Mar 1999 21:12:38 -0800
To: discuss@apps.ietf.org, ietf-http-ng@w3.org
Message-ID: <9903082112.aa09585@paris.ics.uci.edu>
Any and all editorial thoughts would be appreciated -- but please
reply to me personally. It seems flabby to me, and I'm too bleary to
tell if it's too lame or too technical at times... RK

This column:  http://www.ics.uci.edu/~rohit/IEEE-L7-applcore.html
First column: http://www.ics.uci.edu/~rohit/IEEE-L7-1.html
General info: http://www.ics.uci.edu/~rohit/

Seventh Heaven appears in each issue of IEEE Internet Computing:

Rohit Khare -- UC Irvine -- 4K Associates -- +1-(626) 806-7574
http://www.ics.uci.edu/~rohit -- http://xent.ics.uci.edu/~FoRK


   Building the Perfect Beast
   Dreams of a Grand Unified Protocol
   Rohit Khare // March 5, 1999
   IEEE Internet Computing // Seventh Heaven
   Seventh Heaven was inaugurated one year ago to chart the evolution of
   Internet application protocols. Perched upon the heights of the
   network stack, we paged back through history to trace the lineage of
   Telnet, FTP, SMTP, NNTP, Gopher, and HTTP. With Jon Postel's
   fingerprints to guide us, the arc of thirty years of Transfer Protocol
   (TP) development brought us to the present day -- where the pages of
   our history books go blank.
   Leaving us, gentle reader, with but one option for our second season:
   to turn from documenting the past to divining the future. After all,
   unlike designs past, there are infinite number of designs yet-to-be:
   and every other acrynomious concoction we can get our hands on(*). And
   that's to say nothing of how protocols are actually adopted and
   history actually gets written: the mythical eighth (Economic) and
   ninth (Political) layers of the stack. These are the kinds of new
   topics we'll tackle in 1999: new protocols, economic models, and
   standardization processes.
   (*) [Footnote: For the record, that's Remote Procedure Calls in
   Extensible Markup Language, HTTP-Next Generation, an extensible
   successor to RADIUS (Remote Dial-In USer) authentication, Instant
   Messaging and Presence Protocol, Session Invitation Protocol,
   Internet-Scale Event Notification, Simple Workflow Access Protocol,
   Wireless Access Protocol, and the Secure Digital Music Initiative.]
   Johnny APPLCORE
   Conveniently, this raises a very timely topic to the head of our
   agenda: the IETF Applications Directorate's debate over the
   possibility and desirability of an Application Core Protocol
   (APPLCORE). This latest outbreak of an ancient meme bloomed rapidly,
   from its first mention in January to a Birds-of-a-Feather session at
   IETF-44 in Minneapolis by March. The proliferation of new applications
   derived -- or worse, cut-and-pasted by parts -- from SMTP and IMAP
   (Interactive Mail Access Protocol) has prompted an interest in
   documenting the common challenges of such extensions, and perhaps in
   developing an entirely new set of command building-blocks.
   Another variant of the same infectious hope motivates the HTTP-Next
   Generation project, sponsored by the World Wide Web Consortium with
   participation from Xerox and Digital (profiled in detail by Bill
   Janssen last issue). From their vantage point, the Grail looks like a
   programmable distributed object protocol, atop an efficient new
   multiplexing transport. They look ahead to interoperating with Java
   RMI, CORBA IIOP, and RPC protocols.
   And of course, HTTP/1.1 itself has proven flexible enough for a host
   of communities to (ab)use it to their own ends. From distributed
   authoring and versioning (WebDAV) to the Internet Printing Protocol
   (IPP), the freedom to define new methods, headers, and Internet Media
   Types at will has led to at least three schools of thought on
   extending the hypertext Web to other information transfer problems.
   Its proposed Mandatory extension mechanism aims to rationalize the
   patchwork of existing rules.
   These three camps -- Mail, Web, and Objects -- are each vying to
   establish a new order for Application Layer Protocol design. The quest
   for universal applicability, though, crosses several common fault
   lines in the design space. Each camp tends towards different tradeoffs
   between human- and machine-readability, command-oriented and stateless
   transactions, end-to-end and proxiable connections, inspectability and
   security, and whether to exploit the unique properties of UDP,
   multicast, broadcast, and anycast transmission.
   While it may seem quixotic to hope for a Grand Unified Protocol, part
   of the IETF ethos is that you never know unless you try. However slim
   the opportunity for clean reengineering, it is still appears
   worthwhile to each camp to seek out an "application-layer TCP," a
   second neck in the Internet hourglass.
   Or, as mail protocol and security maven Chris Newman put it when
   calling for APPLCORE in the first pace, "What protocol facilities are
   common to most of FTP, HTTP, IMAP, LDAP, NNTP, POP, SMTP/ESMTP, Telnet
   and our other successful protocols? ... I can't predict what a
   careful study of IETF protocol history and comparison of candidate
   solutions will suggest." Since that's the very mission statement of
   our column, let's take a shot at evaluating these camps'
   Diversity vs. the Melting Pot
   While "transport" protocol merely deliver piles of bits, TPs need to
   'tag-and-bag' their information, almost invariably using MIME syntax.
   IETF's eventual success in hammering out a standard for packaging
   information objects is a powerful inspiration for today's protocol
   It's not much of a stretch to think of today's TPs as merely different
   strategies for delivering MIME objects -- real-time vs batch, push vs.
   pull, end-to-end vs. relayed, reliable vs. unreliable, and so on. Over
   the past year, this column has fleshed out a model of Transfer
   Protocols (TPs), as summarized in Table 1. We classify three aspects
   of TPs: addresses that identify participating nodes; distribution
   rules controlling transfers; and the message formats on the wire.
   Before proposing a universal solution by picking winners and losers,
   it behooves us to investigate the tensions and tradeoffs that
   motivated such diverse solutions in the first place.
   Engineering for Humans or Machines?
   While all of these protocols are executed by machines, they are
   created by programmers -- two actors with rather different priorities
   for standardization. Will the messages be binary (and faster for
   computers to parse) or English text (and easier for humans to debug
   and extend)? Will the application semantics be precisely defined, a la
   RPC, or open to extensible interpretation, like HTTP?
   Syntax. Two archetypal alternatives are RFC822 header fields and X
   Window System events. Email, news, and web headers from any
   information service are reasonably easy to understand, at the cost of
   marshaling complex data structures like dates and numbers as ASCII
   text and parsing all sorts of variations of case, line folding,
   comments, character sets and other gremlins to uphold the maxim "be
   liberal in what you accept". X Protocol messages are tightly packed
   binary data aligned on machine boundaries -- and can be processed
   several orders of magnitude more efficiently because the control flow
   through a server is clearly delineated by X Window System semantics
   and a strict extension mechanism.
   Furthermore, both alternatives are relatively brittle.
   Internationalizing header text in various human languages is
   notoriously difficult because of the number of techniques available
   (RFC 2277, "IETF Policy on Character Sets and Languages"). IETF also
   has some (exasperating) experience with ISO's Abstract Syntax Notation
   (ASN.1) self-describing binary marshaling format for SNMP and X.500
   public-key certificates. The tools for compiling such data structures
   are expensive and still can't guarantee interoperability amongst all
   the possible Encoding Rules (Basic, Defined, Text, ...).
   The silver bullet of the moment is XML. Roll-your-own tagsets offer
   some hope of jointly guaranteeing machine-readable validity and
   human-readable message text. As WebDAV encountered, using XML in a
   wire protocol has its own costs: complex machinery for XML text
   encoding ("entities"), whitespace, namespaces, and bloated message
   text. And for all that, there's still no standard data-formatting
   rules yet for basic data types like dates, numbers, arrays, and so on.
   The ability to simulate a protocol exchange with Telnet is another
   litmus test whether it's catering to machines or humans. If the
   application itself expects to be supported by individual users and
   system administrators, it's useful to experiment and debug the system
   with a bare-minimum tool available on almost any host. If it's a core
   OS service supported by the vendor, then it's reasonable to expect
   libraries for inspecting binary messages and APIs to support it.
   Semantics. Another way of justifying the "Telnet test" is to estimate
   how many independent implementations are expected. While there are
   only a few X servers and client libraries -- including definitive
   editions from the X Consortium -- there are uncountable little
   HTTP-driven scripts and hacked applications babbling away in pidgin
   dialects developers learned by imitating other clients and servers.
   The smaller the community of developers, the greater the likelihood of
   establishing a common ontology. A system as multilateral as the Web
   dilutes the semantics of "GET" to the point it could be opening a
   database, running a Turing-complete program, instructing a robot, or
   any other process that eventually generates a MIME entity.
   Turn the argument on its head, and it illuminates the tension between
   APIs and protocols. If there is a suite of clearly defined operations,
   there's more ease-of-reuse by standardizing programming interfaces
   ("the Microsoft way," Generic Security Services (GSSAPI, RFC 1508)).
   If implementations are expected to diverge, just focus on the
   bytes-on-the-wire and the message sequence ("the IETF way," Simple
   Authentication and Security Layer (SASL, RFC 2222)).
   It's the Latency, Stupid!
   Performance constraints also vary with human-driven or machine-driven
   transactions. Interactive use must minimize latency, leading to
   stateless protocols, while batched server-to-server communications can
   optimize bandwidth utilization with stateful command protocols.
   In the messaging arena, POP and IMAP provide client-server access,
   while SMTP and NNTP relay between stores. POP optimizes latency by
   selectively listing headers and bulk data separately; IMAP further
   offers concurrent operations. SMTP and NNTP, though, operate in modal
   lockstep, pipelining transmission and reception of new messages.
   On the Web, last issue's column suggested that the earliest HTTP/0.9
   spec was even less powerful than FTP -- but it had a critical
   advantage of lower latency. While FTP requires separate commands (and
   round-trips) to login, authenticate, navigate to a path, and request
   transmission, an HTTP download begins within one round-trip after
   connection establishment. That's because its request message
   encapsulates all those commands in a single message (URLs for path and
   filename; headers for authentication information and media-type) --
   and each request can be processed on its own, without reference to the
   state of a connection.
   As the 'bandwidth-delay product' gets larger and larger for wireless,
   fiber, and satellite links, stateless protocols become more valuable
   -- it takes less time to pickle the added state information and
   transmit it than to wait for a command to complete.
   Relay Races
   End-to-end support is another contentious design decision facing our
   conquering heroes. Stateful command sequences are harder to proxy
   through firewalls or cache, but can manage concurrency explicitly.
   Store-and-forward messaging (TPs) typically operate across a chain of
   relays, in contrast to interactive query and access protocols (APs)
   directly connecting clients and servers.
   Consider, then, the Calendaring and Scheduling WG's dilemma. Their
   requirements included connected and disconnected operation, queries
   against multiple stores, and low-bandwidth operation
   (draft-ietf-calsch-capreq-02.txt). There are some arguments for
   modeling a calendar as a Web resources, and operations upon it as HTTP
   transactions. In particular, the DAV and DAV Searching and Locating
   (DASL) extensions cover some of their requirement space. It's unclear,
   though, if the HTTP caching model is sufficient for nomadic use,
   though. An access protocol like IMAP has richer support for reflecting
   local actions immediately, chains of actions, and conflict resolution,
   but at the expense of custom development (about four years in this
   Concurrency is also represented differently in each school. An AP can
   tag its requests by ID and thus reshuffle its responses to several
   outstanding operations in the same session. APs like SMTP and NNTP can
   also "turn" the connection around to make requests in the opposite
   direction. Stateless TPs typically require synchronous response,
   tacitly pushing concurrency control -- the timing and priority of
   responses -- to the transport layer. Thus, Web browsers that open
   multiple TCP connections; and the Message Multiplexing (MEMUX) effort
   within HTTP-NG.
   Unlike end-to-end APs, proxying permits intermediaries to offer
   sophisticated services, from caching to Japanese translation. While
   HTTP cannot explicitly model the side-effects of a chain of operations
   (does the server reply to the outstanding GET or PUT first?), its
   statelessness does enable a rich caching model (I don't care as long
   as the reply's no older than five minutes).
   Checkpoint Charlie
   Firewalls are another kind of proxy. Network administrators have a
   right to inspect the contents of Internet connections. Today's
   baseline is filtering services by TCP port number, but as more and
   more applications attempt to extend HTTP or other APs, the port number
   isn't precise enough to enable or disable specific services. It's
   intellectually dishonest to use an existing, popular protocol as mere
   transport "because it gets through firewalls" -- they're there for a
   For example, one totally flexible universal protocol is to map remote
   procedure calls to HTTP, whether in XML or directly coding, say
   Distributed COM (shipped with NT5, though thankfully off by default).
   If Web traffic could now conceal information leaking out or hackers
   coming in...
   Even multiplexing makes firewalls more resource-intensive. One TCP
   connection running MEMUX could have several subchannels, each of which
   must be judged individually.
   End-to-end encryption of an AP is another tough scenario: while a TP
   proxy might be able to decrypt and then forward an entire information
   object, it can be a violation of the standard to interpose a proxy in
   the middle of a stateful conversation.
   Security is more than transport-layer encryption alone. There are far
   too many application-layer Authentication and Authorization schemes.
   SASL succeeded by providing a simple building block -- a sequence of
   challenge-response messages and status codes -- that allow developers
   to mix-and-match authentication algorithms.
   Making the Medium the Message
   The layers below also offer application designers unique capabilities
   often overlooked. Telnet, for example, relied on TCP's urgent data and
   interrupt facilities, but HTTP serenely floats atop any 8-bit clean
   channel (even half-duplex!). Since there are so few applications that
   take advantage of broadcast, multicast, and anycast semantics, it's
   hard to plan ahead for a core protocol that could bridge those modes.
   And even though most services reserve TCP and UDP ports, datagrams are
   typically used for "small enough" messages -- it's the rare protocol
   that intelligently copes with lost packets.
   Link-layers also affect the evolution of application protocols. The
   Wireless Access Protocol suite is founded on a belief that every
   Internet layer must be reinvented for the cellular environment.
   Very-low-bandwidth and very-high-latency environments call for compact
   message encoding and pipelining, among other features.
   Camp the first: Mail
   The initial call for APPLCORE came from folks in the email and
   directory communities. A strawman proposal like Application Core
   Protocol (draft-earhart-acp-spec) can trace its heritage back to
   Postel's SMTP and FTP state machines and his theory of reply codes. It
   defines a framework for transitioning to an authenticated connection,
   issuing commands, and receiving interleaved, tagged responses a la
   While it's easy to imagine, say, NNTP's current-group and
   current-article pointers and commands in this vein, the APPLCORE
   charter per se does not call for a stateful solution. It's just a
   coincidence that its author believes the proposed WG should " focus on
   a single "core protocol" based on the connection-based stateful
   client-server structure that most successful IETF application
   protocols follow."
   Camp the second: Web
   HTTP proponents would beg to differ at Chris Newman's imprecation that
   it's one of the "protocol models with which we have far less
   experience...research problems and thus out-of-scope." They can
   point at a handful of significant, standardized extension packages for
   HTTP/1.1 -- and a massive set of informal experiments. Its stateless,
   textual model is particularly hackable, even by shell scripts and
   programs without any internal model of the Web.
   HTTP/1.1 enables all this by permitting new methods, header fields,
   and content types. WebDAV, for example, modified the PUT method with
   lock fields; added MOVE and COPY (limited by a Depth: header); and
   manipulates metadata with XML-coded requests and responses.
   Complexity is the natural consequence of such freedom. Consider how
   byte-ranges, the ability to request only a portion of the expected
   response (typically, for partial rendering, e.g. a single page of a
   PDF file), interact with all other possible extensions.
   This community's best solution is the Mandatory extension mechanism
   (draft-frystyk-http-mandatory), which provides a namespace to discover
   more about an extension, and switches to indicate whether to succeed
   or fail if an extension isn't recognized. This permits multiple
   extensions to interoperate by marking off their own method names,
   header fields, and error codes. Extensions are identified by URIs and
   marked as Mandatory or Optional obligations on the next server (proxy)
   or only the origin server.
   The risk of continuing to subsume more application services within
   HTTP is the tendency to abuse it as an opaque container for their own
   traffic. One of its authors, Roy Fielding, declaimed: "The Web uses
   HTTP as a transfer protocol, not a transport protocol. HTTP includes
   application semantics and any application that conforms to those
   semantics while using HTTP is also using it as a transfer protocol.
   Those that don't are using it as a transport protocol, which is just a
   waste of bytes."
   Camp the third: Objects
   The third tack is to directly model Internet applications as
   distributed programs, and the client-server protocol follows as a
   consequence of the API. The IETF's standards-track Remote Procedure
   Call v2 specification (RFC 1831) was issued four years ago, based on
   fifteen years' experience at Sun and elsewhere. The popularity of
   CORBA and Java RMI interfaces underscores the power of this approach.
   So while the Web camp could be said to provide OO interfaces through
   document transfer (FORMs and TABLEs and so on), the HTTP-NG effort
   aims to provide document transfer over an OO interface. Its three
   layers begin with MEMUX to optimize transmission for the Web by
   composing compression, encryption, and other services for its virtual
   channels; then a messaging layer that can marshal binary request and
   response messages; and its own upper layer for services like The
   Classic Web Application (TCWA), WebDAV, printing, and so on.
   It's certainly appealing to reuse application protocols by subclassing
   and adding your unique functionality, then composing it with other
   off-the-shelf modules for security and performance. But Chris Newman
   invokes the conventional wisdom at the IETF: "RPC mechanisms are a
   poor choice in general for standards-based protocols. It's much harder
   to design an extensible and simple API than it is to design an
   extensible and simple wire protocol."
   So with each camp set against the others, how likely is any compromise
   core protocol -- or one camp's hegemony? The answers are not at layer
   7 alone: we must climb higher. Technical analysis alone does not point
   to a clear intersection of services, nor justifies any protocol's
   universal utility.
   Layer 8: Economics
   First, investigate whose scarce resources are being reallocated.
   "Reuse" claims litter the 120+ message archive of the debate
   (http://lists.w3.org/Archives/Public/ietf-discuss/) -- but whose
   efforts are being reused?
   There are at least three plausible actors: programmers, spec writers,
   and standards committees. The first group doesn't benefit until
   there's a single core protocol on the wire and a code library -- and
   then only for developing 'multi-protocol' tools. The second benefits
   by consulting the accumulated wisdom of APPLCORE to navigate the
   archives and cite prior art. The third benefits from rigorous modeling
   of the design alternatives and a framework to classify inter-extension
   dependencies in order to judge which protocols ought to be blessed.
   Now we can identify institutions that would be motivated to invest in
   this effort. Look for vendors who implement lots of protocols within a
   single product. Microsoft, for example, promotes a single "Internet
   Information Server" with the vision of vending the same information by
   HTTP, DAV, and other protocols. Netscape has a new app for every
   protocol. Apache, focused purely on HTTP, has little interest in
   implementing printing, conference calls, or other whiz-bang
   applications. In the second group are authors with experience from
   multiple working groups. Such IETF luminaries are leading each camp,
   respective examples such as Chris Newman, Henrik Frystyk Nielsen, and
   Mike Spreitzer.
   Layer 9: Politics
   To identify the third class of actors, though, requires understanding
   motivations at the highest level: Politics. The standardization
   process is an institutional struggle, in this case to establish order
   within the IETF Directorates, and to defend its turf from ISO and
   other bodies.
   APPLCORE intends to "go only in directions that at least two
   successful IETF protocols have gone," Newman claims. This extends the
   "rough consensus" maxim to bless reuse of solutions as guidance to
   spec developers. But to envision and APPLCORE robust enough to measure
   which protocols are worthy or not is another order entirely -- and one
   the IESG may not support.
   For example, SG member Brian Carpenter commented "that approach could
   imply that HTTPng is the basis for all future applications
   protocols...I doubt that the IETF is ready to make that step."
   There are similar interests against defining the Internet as mail-like
   or web-like.
   Instead, if no dominant camp emerges, a compromises which aims for
   their intersection raises the scepter of another bogeyman: ISO. IETF
   culture alternately derides for lowest-common-denominator designs with
   lots of options required for the useful cases; and for wasteful
   layering. Charges of ISOism have been leveled by all sides in this
   debate. Grand Unification Theories from the Object Management Group
   (OMG) don't go over well, either.
   All of these considerations feed into the decision whether IETF even
   charters a WG on this topic. Without viable candidate protocols, it
   smacks of design-by-committee -- which triggers quite an allergic
   reaction in this community.
   But if -- just suppose -- APPLCORE succeeded, what a wonderful world
   it would be! An application designer would only have to focus on
   understanding the underlying interaction pattern. Leave it to this bit
   of middleware to choose the right addressing system, syntax, and
   distribution algorithm in order to get the right bag of bits to the
   right people by the right time. Internet application design seem more
   like "rational drug discovery," adapting a protocol tuned to its
   requirements rather than merely chasing popular implementations and
   patching them to match.
   We expect new kinds of hybrid applications: Event notification with
   server-initiated delivery, not just polling. Smart pages that
   represent live processes. Multiprotocol message archives accessible
   over the web, on the phone, as a fax, by e-mail...
   Sure, it would only be a 'good enough,' but a single interface would
   be immensely valuable. TCP isn't ideally tuned for many applications,
   but its ubiquity has a quality all its own. Its twenty-year dominance
   has relied on technical ingenuity to manage its evolution and explicit
   political commitment. We have IP for packets, TCP for streams, and a
   clear vacancy for a message middleware standard.
   The Perfect Beast
   Realistically, though, we suspect this outbreak of the Perfect Beast
   meme will only take us as far as cataloging design patterns of
   Internet application layer protocols. Thirty years' experience is long
   enough to write a textbook, even if the ultimate solution isn't at
   hand. We can capture common solutions to common problems along with
   the rationale. Guidance like "measure compatibility with capability
   lists rather than version numbers" would greatly reduce design-time
   costs -- and guide repairs to today's protocols.
   It's instructive to reflect upon other outbreaks. Mightn't the search
   for One True Application Protocol lead to the same cul-de-sac as the
   One True Programming Language? That community has also played a game
   of turtles: whether functional, imperative, or procedural ought to be
   the most fundamental representation. LISP has been the language of the
   future since 1959 -- even as it has grown to accommodate procedures
   (Scheme), objects (CLOS), and a vast array of time-tested tools (the
   Common Lisp environment). The risks of sprawling unification were
   incoherence, poor performance, and refragmentation into sublanguages.
   There is a fundamental phenomenon at work: ontology recapitulates
   community. Ask how many people need to understand a given sentence?
   -- whether source code or protocol message. That's how universal
   the solution needs to be. If we want every Internet-accessible
   resources in a global hypermedia system, you can't expect more
   than a snapshot representation; if you expect to manipulate a datebook
   to automatically schedule a meeting, then only special-purpose
   calendaring tools will be cognizant of other conflicts.
   All we can hope for is to make the common case easy and keep the hard
   stuff possible...
    Rohit Khare is a graduate student in the WebSoft group at the
    University of California, Irvine; and a principal of 4K Associates,
    a standards strategy practice.
   [Word count:~4,000..Note to the editors: feel free to elide or
   substitute the intra-document headings Note: Table 1 can be copied
   directly from the March/April 1998 issue's Table 1, with a suitable
   change of tense in the caption.]
     CAPTION: Three key aspects of the Transfer Protocols (TPs) Seventh
                           Heaven covered in 1998
Transfer Protocol	Addressing	Distribution	Content
- ------------------------------------------------------------------------

Terminal Telnet	Host	Port	1-1	Sync	Both	Bytestream w/interrupts
Files	FTP	Host	Path	1-1	Sync	Both	Text / Binary Files
E-mail	SMTP	Mailbox	Msgid	1-N	Async	Push	822 + MIME
Usenet	NNTP	Newsgroup	Msgid	N-N	Async	Push	822 + MIME
Web	HTTP	Host	URL Path	1-N	Sync	Pull	822 + MIME + 
                                                                HTTP caching


- ------- End of Failed Message

------- End of Forwarded Message
Received on Tuesday, 9 March 1999 00:15:34 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:38:00 UTC