[Editorial Draft] Extending and Versioning Languages: Compatibility Strategies

1.1 Terminology

[Definition: A Language consists of a set of texts and a mapping between texts and information.]

[Definition: A language is Extensible if the syntax of a language allows information that is not defined in the current version of the language and provides default handling of documents in any extended set.]

[Definition: Every language has a Defined Text set, which contains only Texts that contain the texts explicitly defined by the language syntax constraints. ]Typically, a language will define a mapping from each of the definitions to information conveyed by instances of those definitions. [Definition: Each language has an Accept Text set, which contains texts that are allowed by the language constraints]. Typically, the Accept Text set contains Texts that are not in the Defined Text set.

[Definition: Language L2 is backwards compatible with Language L1 if L2 Accept Text set > (superset) L1 Defined Text Set AND every text in L1 Defined Text set is compatible with L2.]

[Definition: Language L1 is forwards compatible with Language L2 if L1 Accept Text set > (superset) Language L2 Defined Text set AND every text in L2 Defined Text set is compatible with L1.]

When languages are evolved by extending the Defined Text set, forwards and backwards compatibility can be combined for compatibility: [Definition: Language L2 is compatible with Language L1 if L1 Accept Text set > (superset) Language L2 Accept Text set > (superset) L2 Defined Text set > (superset) L1 Defined Text Set AND every text in L1 Defined Text set is compatible with L2 AND every text in L2 Defined Text set is compatible with L1.]

1.2 Why Do Languages Change?

There are many reasons why a language may evolve. A few of them include:

1. Bugs may need to be fixed. Production use may reveal defects or oversights that need to be fixed. This may involve changes to texts of the language or changes to the information of existing texts.

2. Changing requirements may motivate changes in the language. For example, a person name structure may be extended with a middle name, prefix, suffix, and/or common name.

3. Different variations of a language may be desirable. For example, the XHTML 1.0 Recommendation defines strict, transitional, and frameset languages. Although the three forms use markup from the same namespace; strict, transitional and frameset are different languages. Additional languages may be defined by other specifications, such as the XHTML Basic Recommendation.

Whether ten, a hundred, or a million applications have been deployed, if a language is changed in such a way that those applications will determine that texts of the new language are invalid, a versioning problem with real costs has been introduced. Whatever the cause, over time, different versions of the language exist and designing applications using it to deal with the changes in a predictable, useful way will require a versioning strategy.

1.3 Kinds of Languages

There are different kinds of languages and the versioning approaches and strategies that are appropriate for one kind of language may not be appropriate for another. Some of the various kinds of languages are:

• Just strings: some languages are just set(s) of strings. Strings that identify countries, states or provinces, airport 3 letter codes, and traffic light colors are examples of "just string" languages.

• Non-markup Text: languages designed with a character-based text format. These may be programming languages such as Java or ECMAScript, or data formats like CSS or Comma Separated Values. Typically these are intended for humans to author and/or view.

• Markup: SGML, HTML, and the non-SGML variants of HTML are all character based markup languages. Versioning XML languages is described in Versioning: XML

• binary: languages that are not in a text format. These may be image formats like GIF, JPEG, or even binary encoded XML.

This is by no means an exhaustive or exclusive list. Many languages actually define multiple kinds or formats of languages. For example, XQuery defines a non-XML language and an XML language.

Languages may be composed of sub languages. If a language is composed of other languages, then languages are considered nested. There may be different versioning strategies for each nested language, and they all combine together into the overall versioning strategy.

2.1 Why Have a Strategy?

Different kinds of languages and different versioning strategies expose different problems. Attempting to deploy a system that provides no versioning mechanism may put the burden of version "discovery" on consumers and my be impractical in anything except a closed system.

At the other end of the spectrum is an incompatible versioning approach which is also problematic. "Incompatible" versioning is usually a very coarse-grained approach to versioning. It often requires that all of the text must be understood and known by the consumer or the consumer will fault. It often establishes a single version identifier, such as a version number or namespace name, for an entire text. In many cases, a single version identifier is not an accurate representation of the "version" of the text, particularly when a text is composed of many sub-languages.

The semantics of the incompatible approach are that applications decide on the basis of the version(s) of the language they understand whether they can process that text. If parts or all of the text isn't recognized, perhaps including a version identifier, then the entire text is rejected. Typically, when introducing a new version using the incompatible approach without any provisions for backwards or forwards compatibility, any of the software that produces or consumes the texts cannot successfully communicate until it is updated and deployed. This incompatible approach to versioning is practical only in circumstances where there is a single controlling authority, and even in that case, it carries with it many problems. The process can take a considerable amount of time, leaving parts of the system out of commission for hours if not days. This can result in significant losses if the system is a key component of a revenue generating business process and the cost of coordinating the system overhaul can also be quite costly as well.

The incompatible approach is appropriate when the benefits of introducing the change outweigh the costs, such as when the new version is radically different from its predecessor. But in many cases, changes are incremental and often a consumer could cope with the new version. For example, it might be that there are many texts that don't use any new features from the new version or perhaps the unrecognized parts of the text could be ignored.

Consider a producer and a consumer exchanging texts of a particular language. Imagine that some future version of the language includes new texts in the language. Because producers and consumers are distributed, it may happen that an old consumer, one unprepared for the new text, encounters a newer text sent by a newer producer.

If incompatible versioning is used, old consumers will reject the new text. However, if the versioning strategy allows the old consumer to process the text even with the unrecognized content, it's quite possible that other components of the system would adapt to the previous behavior. In effect, the old system would ignore the new component or provide some other default interpretation for it. For the producer, the result would be that the request is fulfilled, though perhaps not taking into account the new features. For example, a request that results in a response may return a text of the language without the new component. In many cases this may be better behavior than receiving an error. In particular, producers using the new language can be written to cope with the possibility that they will be communicating with older consumers.

If the producer needs to require that the new text is understood, then it can change the language in a way to indicate that the new text is not considered optional, aka incompatible.

Often, what is needed is some sort of middle ground solution where compatible and incompatible changes are supported, and these are discussed in the sections on compatible and incompatible.

<name version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<span class="fn20">Dave Orchard</span>

urn:nameschemev2:given:Dave:family:Orchard

<?XML version="1.1"?>

GET /name/123456789  HTTP/1.1

GET /name/v2/123456789/ HTTP 1.1

4.1 Replacement

In the replacement design, a new version of software replaces the old and the new version of the software supports the old and the new version. The producer may or may not need to distinguish between the old and the newer consumer or the texts produced. For example, a web resource that supports additional Name Information as input may not need to change the URI of the resource.

As our definition of backwards compatibility specifies that the newer language's Defined Text Set must be a superset of the older language's Defined Text Set, the typical change is the addition of optional content into the newer version of the language. The older producer simply won't produce texts with the newer content. It is possible to reduce the Defined Text Set by removing items and achieve backwards compatibility, as long as the newer Language's Accept Text set contains all the texts originally in the Defined Text Set. One mechanism to do this is to replace the content with a construct that allows the removed construct. For example, an element can be replaced with a construct (such as an XML schema wildcard) that allows the element.

An example of a replacement strategy is where dereferencing a URI returns an XML document with a person name that contains family and given names and is specified by an XML Schema. Version 2 of the person name adds optional middle names to the name. This new person name language can be deployed as part of the same URI because the V1 and the V2 person name producers can use the same consumer at the previously defined URI.

4.2 Side-by-side

In the side-by-side design, the new version of the software and the old version of the software are deployed "side-by-side". One variant of the approach is offering both versions of the system, for example by using different URIs for the old and new with a particular focus on enabling older versions of applications to operate on inputs that make use of newer language features. The request to one resource gets mapped to the other resource behind the scenes using a proxy or gateway. This "alternative" approach works when the intermediary can completely handle or generate the new information (for backwards compatibility) or accept the new information (for forwards compatibility). For example, adding an optional element to a data type and deploying to a new URI is suppored because a Web server can typically handle mapping the newer URI to the older URI or mapping the older URI to the newer URI. If both URIs are maintained, then the addition is a compatible change. Another example is where new information is required, such as the priority, and the intermediary can apply a default value to provide the required priority. However, this too has its costs as multiple versions of the software must be supported and maintained over time and there is the added cost of developing the proxy or gateway between the two environments. Further, this does not work in scenarios where the intermediary cannot generate the new required content. For example, if a middle name is required in V2, a middle cannot be generated from just a family and a given name.

5.1 Must Accept Unknowns

A language whose syntax allows additional unknown text also requires a specification of what happens when a text contains such additional and unknown text. By the definition of Extensibility, there must be a default rule for interpreting any additional unknown text. If the extensibility is used in a forwards-compatible way, then by definition the software consuming the extension does not know about the extension and we call such extension an unknown extension. The behavior of software when it encounters an unknown extension should be clear.

The simplest model that enables forwards-compatible changes is to require that a language consumer must accept content that is unknown. This rule is:

Good Practice

Must Accept Unknowns Rule: Consumers MUST accept text portions that they do not recognize where the language has allowed extensibility.

HTML 4.01 follows this approach in "If a user agent encounters an element it does not recognize, it should try to render the element's content". The Must Accept Unknowns rule for XML was first standardized in the WebDAV specification RFC 2518 [6] section 14 and later separately published as the Flexible XML Processing Profile [3].

An extension that affects an existing component in an incompatible way is an incompatible change because a consumer that is unaware of the extension will produce Information from the text that is incompatible with the intended Information. An additional rule is required:

Good Practice

Preserve existing information Rule: An Extensible Language SHOULD require that any texts with extensions SHOULD be processible as a text without the extensions.

An example of an incompatible extension because of a violation of this rule is a purchase order with a payment amount that is in US dollars that is extended by an element that specifies the payment amount is in Euros. An older consumer that ignores the extension element will incorrectly assume that the payment amount is still in US dollars when in fact it is in Euros.

There are two commonly deployed refinements of the Must Accept rule that qualify what kind of handling beyond accepting is required.

Must Accept and Discard Unknowns Rule: Consumers SHOULD accept and discard any text portion that they do not recognize. This is commonly shortened to "Must Ignore Unknowns Rule". HTML 1, 2 and 3.2 follow the Must Accept and Discard Unknowns Rule as they specify that any unknown start tags or end tags are mapped to nothing during tokenization.

Discarding content is a simple solution for the default processing rule for unknown content. In order to achieve a compatible evolution, the newer texts of a language must be able to be treated as older texts when the unknown content is discarded.

Another model is to preserve the unknown. Must Accept and Preserve Unknowns Rule: Consumers SHOULD accept and preserve any text portion that they do not recognize. HTTP 1.1 [7] specifies that a transparent proxy should accept and preserve any headers it doesn't understand: "Unrecognized header fields SHOULD be ignored by the recipient and MUST be forwarded by transparent proxies."

Many languages are consumed in multiple stages, each of which may have different variants of these models. An example is a web browser processing HTML. It will typically tokenize HTML first, and HTML specifies that unknown markup is tokenized to nothing. The results of tokenization are used as the basis for rendering. The same markup is still placed in the DOM and is available for CSS or other DOM related technologies. Another way of looking at this combination is that there are two languages. The browser tokenizer understands HTML which involves ignoring unknown elements or attributes. By our language definitions, the renderer's Defined Text set does not include unknown elements or attributes, though the Accept Text Set does. The browser DOM understands any HTML elements or attributes. The DOM Defined and Accept Text set includes any elements or attributes.

5.2 Fallback Provided

A language can provide mechanisms for explicit fallback if the text is not supported. [MIME] provides multipart/alternative for equivalent, and hence fallback, representations of content. [HTML 4.0] uses this approach in the IMG element where the alt attribute is used if images are not supported or disabled. In XML, the XML Inclusions specification [XInclude] provides a fallback element to handle the case where the putatively included resource cannot be retrieved. There are many variations on where the fallback content can be found. For example, a schema language could specify that fallback content is found in a text, in a schema, or even in the schema for the schema language.

There can be difficulties with fallbacks when the consumer chooses one alternative, modifies it and then a subsequent or different consumer chooses a different alternative. In the multipart/alternative example, an email could be offered as HTML and plain text. If the consumer chooses plain text, replies (becoming a producer) with in-line quoting, then a next consumer chooses to reply to the new message using HTML, then the alternatives are not synchronized and are not true alternatives any more.

5.3 Version identifiers

Providing forwards compatibility often requires more than a processing model for texts with unknown content, it must also provide a model for version identifiers. Two of more the popular ones follow.

5.4 Supporting functionality

Additional functionality can be provided in a language for determining the capabilities of the system that the text is being interpreted in. A language can provide a mechanism for explicit testing, and if so, the language also needs a mechanism for the text to contain a ??? for when the test returns false. The XSLT Specification provides a conditional logic element and a function to test for the existence of extension functions. This allows designers of stylesheets to deal with different consumer capabilities in an explicit fashion.