[Editorial Draft] Extending and Versioning w/Noah Aug 27 Comments

0 Overall Comments from Noah

Congratulations, Dave. I think this represents real progress, and obviously a lot of hard work. I think there's a lot of good stuff coming together in this, but I'm afraid I still feel that there significant areas in which either the analysis or its presentation doesn't seem right to me. During one of our telcon reviews, I said that I thought a certain amount of relayering and change of terminology would be beneficial. Dan Connolly quite reasonable said: "well, to justify that, you really should document your concerns with the draft as it stands", and I think that's a fair request. So, this is my attempt to point out in detail some of the concerns that I've had with the finding (though I also try to point out some of the many things that I like; I think it's tremendously improved from earlier drafts.) In some cases, I think the fix is just a bit of tuneup to make the wording more precise. In other cases I think the concerns are somewhat more fundamental, but I don't think the fixes will make the document longer or more complicated. On the contrary, my intuition is that once we get the layering and terminology straight, the story gets both easier and more precise.

Detailed comments are sprinkled throught the first half of the document, which is all I've managed to review so far. These should help to highlight particular areas that I think need work. For your convenience, this introductory section summarizes the highlights of my suggestions and/or concerns:

The finding claims that constraints are part of the language. I'm not convinced that's a good formulation, since the constraints are embodied in the set of texts & mappings. Stated differently, I think we're confusing a "language" with "the specification of a lanuage", and those are very different. So, I think a language should be a set of texts and their interpretation as information, and I am very happy with the way you present that much.

I think we should have separate sections that talk about managing the specifications for languages as they evolve, and certainly constraint languages like XML Schema are among the good tools for writing specifications. It's OK to talk about keeping a language and its specification in sync. and to talk about constraint language features that facilitate versioning. I don't think the constraints are the language. I think they are emergent properties of the language that can sometimes be usefully set down in mathematical and/or machine readable notation such as regex's or XML Schemas. This is an important distinction on which I disagree with the finding as drafted.
I think we can and should do better in telling a story about whether a particular text is compatible as interpreted in L1 or L2, vs. the senses in which languages L1 and L2 as a whole are compatible. I think the story I would tell would be along the lines of:
- Of a particular text written per language L1 and interpreted per language L2: "Let I1 be the information conveyed by Text T per language L1. Text T is "fully compatible" with language L2 if and only if when interpreted per language L2 to yield I2, I1 is the same as I2. Text T is "incompatible" if any of the information in I2 is wrong (I.e. was not present in I1 or replaces a value in I1 with a different one...this rules disallows additional information, because only the information in I1 is what the sender thought they were conveying, so anything else is at best correct accidently). There are also intermediate notions of compatibility: e.g. it may be that all of the information in I2 is correct, but that I2 is a subset of I1. [Not sure whether we should name some of these intermediate flavors, but if we do, they should be defined precisely.]
- Of languages L1 and L2: We say that language L2 is "fully backward compatible" with L1 if every text in L1 is fully compatible with L2. We say that language L1 is "backwards incompatible" with L2 if any text in L1 is incompatible with L2. We say that Language L1 is "fully forwards compatible" with L2 if every text in L2 is fully compatible with L1. We say that L2 is "forwards incompatible with L1" if any text in L2 is incompatible with L1. As with texts, there may be intermediate notions of langauge compatibility for which we do not [or maybe we should?] provide names here.
That all seems pretty simple and clean to me, and I think it's a firm foundation for much of the rest of the analysis. Notice that it seems natural to leave out discussion of the constraints in this layer; the story gets simpler without them. The current draft seems to me a bit loose in both talking about and defining issues for languages as a whole vs. for individual texts.
Clarify focus on texts vs. documents

I think the finding should mostly focus on "text(s)" as opposed to "documents". Regarding documents, about all I'd say is:

"This finding describes a language as a set of texts and their associated mappings to information. Each text is an ordered collection of characters (i.e. a "character string"). For the most part, the concepts described here apply both to languages that are used for entire documents (e.g. computer files, messages sent as characters) or to sub-languages that may be created for use within larger texts (a language for representing phone numbers, a language for representing street addresses, etc). Except where the distinction between documents and other texts is important, this finding refers only to "texts", "sets of texts" and so on."

[And optionally: "Indeed, when sublanguages are used, the versioning story is applied recursively: if a purchase order language uses a sub-language for its street addresses, then the compatiblity of the purchase order as a whole is derived in part from the compatibility of the address sub-language; specifically, the address syntax becomes part of the overall purchase order syntax, and the information conveyed for addresses is available when processing orders. Note that, in the particular case of XML, it is common for such sublanguages to have as their texts namespace-qualified elements, such as <addr:Address>, and for such sub-languages to be shared across organizations."]

I think the above shows how, when we get the abstractions just right, things start to fall out. We provide a general story for languages which are texts (not documents), and the hook into XML elements comes almost for free. Indeed, this reminds me that I think we would do better to start with non-XML examples like phone numbers, and then show how a run of text representing an XML element can be a very interesting special case.
I think the finding needs to better separate the compatibility of language versions from the compatibility of various applications that consume those languages.

The draft is on pretty firm ground when it talks about the information that can be determined from a given input text per some particular language L. I think there are important compatibility statements we can and should make at just that level (see suggestions above), and we should separate them from statements about the compatibility of a particular pair of applications that may communicate using the language. Both are important to include, I think, but they should be in separate chapters, one building on the other. Once you've cleanly told a story about which information can be reliably communicated when sender and receiver interpret using different language versions, you can go on to tell a separate story about whether the applications can indeed work well together. To illustrate what I mean, here are examples at each of the two levels.
- Language level incompatibility: Consider a situation in which the same input connotes different information in one version of a language or another. Without reference to any particular application, we can say that the languages are in that respect incompatible. For example, we might imagine a version of a language in which array indexing is 1-based, and a later version in which 0-based indexing is used; the information conveyed by any particular array reference is clearly in some sense incompatible, regardless of the consuming application's needs.
- Application-level incompatibility: Now consider two applications designed render the same version of the HTML language. The same tags are supported, with the same layout semantics, etc. One of the applications, however, has a sub-optimal design. Its layout engine has overhead that grows geometrically with the number of layout elements. If you give it a table with 50 rows, it takes 3 seconds to run on some procesor. If you give it a table with 5000 rows it runs for 3 days. Question: is the second application "compatible" with the 5000 row input? In some ways yes, and in some no. It will eventually produce the correct output, but in practice a user would consider it incompatible. This illustrates that compatibility of applications ultimately has to be documented in terms meaningful to the applications. In this case, rendering time is an issue. I think we should not try in this finding to document specific levels of compatibility at the application level and we should especially not fall into the trap of trying to claim it's a Boolean compatible/incompatible relation; in the performance example, it's a matter of degree. So, the terminology needs to be specific to the application and its domain. I do think we can talk about some meta-mechanisms that work at the application level, such as mustUnderstand, but they should be in a section that's separate from the exposition of texts, information, and the degree to which information may be safely extracted from a given text when sender and receiver operate under differing specifications.
  
  The current draft tries to take the approach that we will model application compatibility by defining a new language that is the flavor of (in this case HTML) that a particular consumer will successfully process, but the point is that "success" is sometimes a fuzzy concept. Do we have two languages for this example, one for the documents that completely break application #2 and another for those that just make it run slowly? That seems to be what the finding is doing today, and I'm not convinced it's the right approach. My proposal would be that we just point out the distinction and say: "This first section of the finding for the most part restricts its analysis to the limited question of: what information can be reliably conveyed when a producer and a consumer operate using different versions of what purport to be the same or similar languages? The later sections explore some techniques that can be used by applications to negotiate means of safe interoperation when sender and receiver are written to differing versions of a language specification."
Introducing mustUnderstand: who sets application-level compatibility expectations, the producer, consumer or both together?

I think mustUnderstand should be discussed primarily at the application level, with the one exception that the mustUnderstand construct itself must be recognized at the information set level. (If I send you an mU element, and you don't even recognize what it means, we're incompatible at the information transfer level; if you recognize it and decide whether to throw an "I don't understand" error for some particular construct, e.g. a SOAP header asking you to start a transaction, when you know you don't have transaction capability, that's an application-level compatibility operation.)

I think it's also interesting to talk about mU in terms of who gets to control application compatibility requirements, the sender or the reciever. Asking that question gives us a really nice entry into discussing mechanisms like mustUnderstand (mU). Basically, mU is a mechanism that allows a sender to establish expectations for success. Using mU=true, the receiver is breaking the rules if it doesn't honor the sender's definition of compatibility. Conversely, with mU=false, the sender is explicitly delegating to the receiver the option to decide what input is good enough to process. Finally, in a system where the mU (or similar) mechanism doesn't exist at all, there may be other means used to establish what represents a successful communication, and hence which message texts are compatible. For example, it's common to use mechanisms like WSDL to establish out of band agreements in advance. In such systems, a bug free sender will never (unintentionally) send an incompatible message to a receiver.
Syntax is not completely separate from semantics or information

The finding should highlight that among the mappings available from syntax to information is the 1:1 mapping. Stated differently the bits that are the transfer syntax, and their interpretation per specifications like Unicode or XML 1.x, are information that is always available at the receiver. So, for example, when you receive an XML document, you can always tell if you want to whether a given attribute is quoted with single (') or double (") quotes. While it's true that for many purposes the information mapping of interest does not carry that information (e.g. the Infoset does not), that information is in principle there. Similarly you can, if you want, tell whether the special <empty-element/> syntax has been used. Indeed, an XML editor that wants to preserve quotes (perhaps so as not to break CVS diffs) will view such information as part of its XML information model. Likewise all the characters may be of interest for a digital signature. So, the syntax is always in principle part of the information; when your mapping drops it that's just a special (though common) case in which you didn't care about it. I think the formulation presented here makes it hard to tell a story about things like HTML editors that work at what one is tempted to view as multiple levels of abstraction simultaneously (knows about the infoset, but also knows about attribute quoting).
Finally, the draft has an unfortunate tendency to say "X is true" when in fact X is only true some of the time, or based on some simplifying assumptions. Fixing these will make the finding more valuable and in many cases is easier to read, precisely because it will convey its points carefully using well chosen words. I've tried to point out some examples in comments sprinkled through the text (including in the first sentence :-) ).

I know that's a lot to raise all at once, but you'll see that I use the above as points of reference in my detailed comments on the draft. So, I hope that the comments below are making the connections Dan and others were looking for. Anyway, I have tried to annotate below specific parts of the draft that I thought either were factually incorrect or would benefit from rework or clarification, and to link to the issues above as appropriate. I hope this is helpful. Again, I appreciate all the hard work Dave is putting into this, and I hope that these suggestions prove useful.

1 Introduction

The inevitable evolution of languages It's not inevitable. There are lots of languages, often simple ones, that don't change at all through their useful life. (e.g. using [0-9]+ in certain systems to represent positive integers). Statements like this are imprecise and therefore invite controversy that distracts from the main points of the document. I think the point you're trying to make is "Many, perhaps most computer languages evolve, but all too often designers fail to plan for such evolution." If that's what you mean, I'd say that (or something similar).by adding, deleting, or changing syntax or semantics is called versioning. Making versioning work in practice is one of the most difficult problems in computing. Arguably, the Web rose dramatically in popularity because the need to deal with evolution and versioning were built intowere anticipated in both HTML and HTTP. Both systems provide explicit extensibility points and rules for understanding extensions that enable their decentralized extension and versioning.

This finding describes general problems and techniques in evolving systems in compatible and incompatible ways. These techniques are designed to allow compatible changes with or without schema propagation. A number of questions, design patterns and rules are discussed with a focus towards enabling versioning in XML vocabularies, making use of XML Namespaces and XML Schema constructs. This includes not only general rules, but also rules for working with languages that provide an extensible container model, notably SOAP.looks like a holdver from before the part 1/part 2 split.

1.1 Terminology

The terminology for describing languages, producers, consumers, information, constraints, syntax, evolvability etc. follows. Let us consider an example. Two or more systems need to exchange name information. Names may not be the perfect choice of example because of internationalization reasons, but it resonates strongly with a very large audience. The Name Language is created to be exchanged. used to communicate names from one application to another. (seems awkward to say that a "language" is created to be exchanged; we don't exchange languages, we exchange texts in the language. Again, I think it pays to be precise.)[Definition: A producer is an agent that creates text. ] Shouldn't you define text first? The definition appears later. Altogether, I think these two paras. should be rearranged so the definitions appear before use. Continuing our example, Fred is a producer of Name Language text. [Definition: An Act of Production is the creation of text.]. A producer produces text for the intent of conveying information. When Fred does the actual creation of the text, that is an act of production. [Definition: A consumer is an agent that consumes text.] We will use Barney and Wilma as consumers of text. [Definition: An Act of Consumption is the processing of text of a language.] Wilma and Barney consume the text separately from each other, each of these being a consumption event. A consumer is impacted by the instance that it consumes. That is, it interprets that instance and bases future processing, in part, on the information that it believes was present in that instance. Text can be consumed many times, by many consumers, and have many different impacts.

[Definition: A Language consists of a set of texttexts [each "text" is a run of characters; the language is the set of such texts, I think, hence plural], any syntactic constraints on the text, a set of information, any semantic constraints on the information, and the mapping between texts and information. ] ***Mostly I like the definition a lot, but the part about constraints seems wrong to me. I'm not convinced either syntactic constraints or semantic constraints are part of the language definition. Please carefully read my comment on constraints above. The syntactic constraints seem redundant, since you've already said that you've got in hand the set of legal texts. Presumably, if a text violates a syntactic constraint, or its mapped information violates a semantic constraint, that text isn't in the set of texts anyway. So, the part I understand is: "A language is a set of texts, and the mapping between each such text and information." The rest I don't get and needs to be better motivated or (I would prefer) we should agree that the constraints are not part of the language definition, but rather are a means of documenting the language definition. [Definition: Text is a specific, discrete sequence of characters]. Given that there are constraints on a language, any particular text may or may not have membership in a language. [In the remainder of the document, you seem to use the terms "text" and "string of characters" interchangeably. You should probably either indicate that you will use both terms for the same thing, explain the difference, or stick to one of them.] Indeed, a particular string of characters may be a member of many languages, and there may be many different strings of characters that are members of a given language. The text of the language are the units of exchange. [You're assuming facts not in evidence. Nothing so far has said that there are any two or more parties exchanging anything. Maybe you should say:] "The texts of a language are often used to exchange information between applications." Documents are texts of a language. See my comment on documents vs. texts above. Maybe or maybe not "document" also needs a formal definition.The Name Language consists of text set that havehas 3 terms and specifies syntactic constraints: that a name consists of a given and a family. [Definition: A language has a set of constraints that apply to the set of strings in the language. ] These constraints can be defined in machine processable syntactic constraint languages such as XML Schema, microformats, human readable textual descriptions such as HTML descriptions, or are embodied in software. Languages may or may not be defined by a schema in any particular schema language. The constraints on a language determine the strings that qualify for membership in the language. Vocabulary terms contribute to the set of strings, but they are not the only source of characters to the set of strings in a given language. The language strings may include characters outside of terms, such as punctuation. One reason for additional characters is to distinguish or separate terms, such as whitespace and markup.

IMPORTANT: As previously noted, I would have thought that the constraints are not part of the definition of the language, but are an emergent property. I strongly believe we should leave discussion of constraints out of this chapter entirely.

I would put in a completely separate (and very important!) chapter on language specifications and on communicating language definitions between organizations. There I might say something like:

["Definition: A language specification is a set of constraints sufficient to convey the definition of a language, I.e. to specify which texts are and are not in the language, and for those that are, their mappings to an information set.] Such language specifications can be used to establish agreement on what a language is and what it means when texts or documents are to be shared among organizations. So, it's common to provide new specifications whenever a language changes. There can be multiple specifications for the same language, perhaps using different specification technologies (such as XML Schema and RelaxNG); of course, such duplicate specifications should generally be consistent. In such cases, we still speak of one language (e.g. HTML 4.0 Strict), but of multiple specifications for it." I would point out that languages like regular expressions, BNF, and XML Schema can be used to directly constrain the legal texts but that sometimes it is more convenient to constrain the texts indirectly by constraining the legal information sets (e.g. a given text is not in the language if it would map to an information set in which max<min).

This new chapter is the one in which you would discuss the need to keep the changing definition of a language in sync with its language specifications and the ways users do or don't share specifications (e.g. schemas) when languages change. You might also say: "Some specification languages, such as XML Schema, have features (such as <xsd:any> wildcards) that are intended to make it easy to specify languages that will easily interoperate across versions."

Anyway, how to slice the chapters is up to you. I strongly feel that all this stuff about constraints and specification language is in a conceptually separate layer, and the current draft has these abstractions somewhat unclearly combined.

Example 1: Name examples.

<name>
  <given>Dave</given>
  <family>Orchard</family>
</name>

name="Dave Orchard" 

<span class="fn">Dave Orchard</span>

urn:nameschem:given:Dave:family:Orchard

The set of information in a language almost always has semantics. In the Name Language, given and family have the semantics of given and family names of people. The language also has the binding from the items in the information set to the text set. Any potential act of interpretation, that is any consumption or production, conveys information from text according to the language's binding. The language is designed for acts of interpretation, that being the purpose of languages. In our example, this mapping is obvious and trivial, but many languages it is not. Two languages may have the exact same strings but different meanings for them. In general, the intended meaning of a vocabulary term is scoped by the language in which the term is found. However, there is often some expectation that terms drawn from a given vocabulary will have a consistent meaning across all languages in which they are used. [Reason: I think the one word insertion is crucial. In some languages the word rod means a cylindrical stick; in some languages it is short for Rodney. There is no expectation that they will mean the same thing. Again, we need to be precise, and I don't think it unduly complicates the presentation to be careful. In this case, we add one word.] Confusion often arises when terms have inconsistent meaning across language. The Name terms might be used in other languages, but it is generally expected that they will still be "the same" in some meaningful sense it is in many places useful and appropriate to use terms such as Name consistently across languages (as we will see later, such reuse becomes easier if mechanisms such as XML namespaces are available; the XML element is in fact naturally used for many conflicting purposes across many XML languages, but we can hope that , where nm: is bound to our naming vocabulary, will be used very consistently in all contexts. (Reasons: (1) Generally expected is too strong -- it's desirable; (2) "Name" is a super bad example of a term that you could expect to have used consistently all across the Web, but with nm:Name the expectation is realistic. Sigh.).

These terms and their relationships are shown below

We say that Fred engages in an Act of Production (can we use that phrase in a W3C document that may be read by minors?) that results a Name Instance (see previous comment) with respect to Name Language V1. The Name Instance is in the set of Name V1 Strings, that is the set of strings in the Name Language V1. The production of the Name Instance has the intent of conveying Information, which we call Information 1. This is shown below:

We say that Barney engages in an Act of Consumption of a Name Instance with respect to Name Language V1. The consumption of the Name Instance has the impact of conveying Information 1. This is shown below:

Versioning is an issue that effects almost all applications eventually. Whether it's a processor styling documents in batch to produce PDF files, Web services engaged in financial transactions, HTML browsers, the language and instances will likely change over time. The versioning policies for a language, particularly whether the language is mutable or immutable, should be specified by the language owner. Versioning is closely related to extensibility as extensible languages may allow different versions of instances than those known by the language designer. Applications may receive versions of a language that they aren't expecting.

If a Name Language V2 exists, with its set of strings and Information set, Wilma may consume the same Name Instance but with respect to the Name Language V2 and have impact of Information 2. Name Language V2 relates to V1 by relationship r2, which is forwards compatible comparing language V1 to V2 instances, and backwards compatible comparing language V2 to V1 instances. I loved most of what you wrote above, but here we need to be careful. Please carefully read my comment on text and language compatibility, This is a central concern for me in working through this draft. In particular, I think we want to be careful about the distinction of a particular text being compatible vs. languages being compatible. Indeed, to some extent, it's hard for me to comment on the rest of this until we come to some agreement on these fundamental abstractions. I suspect that some of the concerns I raised about language-level vs. application-level compatibility may also be pertinent, since it's not clear whether you're worried about all possible Wilmas (in the sense that no matter what application she runs, there would be a compatibility problem because the informatoin received was unreliable) or some particular Wilma (her particular application runs, but maybe too slowly.) In short, I think we need to be much more careful in introducing abstractions of compatibility. Similarly, Information 2 - as conveyed by Consumption 2 - relates to Information 1 - as conveyed by Consumption 1 - by relationship r1.

Extensibility is a property that enables evolvability of software. It is perhaps the biggest contributor to loose coupling in systems as it enables the independent and potentially compatible evolution of languages. Languages are defined to be [Definition: Extensible if the syntax of a language allows information that is not defined in the current version of the language.]. The Name Language is extensible if it can include terms that aren't defined in the language, like a new middle term.

1.1.1 Compatibility

[I hope it's clear by now what my main concerns are in the sections that follows. The following sections introduce compatibility in terms of what processors can and cannot do. I think we need to separate language-level compatibility from processor-level. I think that when we do the core notions of backward and forwards compatibility will not talk about processors, as the sections below do, but will talk about information that is conveyed, information that is lost, and information that is reconstructed in error (I.e. is mapped from the text but is not what the producer intended.) All of this should, I believe, be defined without reference to processors. As noted above, I think we should have a separate section on the sorts of compatibility that particular applications (I.e. processors) may seek, and I expect much of that to be in application-specific terms.

For example, I think the story below about partial understanding needs to be split in that way. First, we tell a story about whether information is lost, or whether information is mapped in some less useful way when transmitted across language versions. For example, V2 of a language may recognize in its information set that the text <middleName>Bob</middleName> has a semantic of being someone's middle name; V1 of the language may just have in its information set "hey, there was a random XML Element <middleName between the <firstName> and the <lastName>; I'm passing it up in the information in case you care" (an example of including syntax in the information set). That story can be told purely at the information mapping level, independent of any particular application. Then you can very nicely say: is particular application A1 happy to get that XML element it wasn't expecting when that newer A2 sent it the middle name? So, a layered story. Seems very clean to me. As I said before, I think mustUnderstand fits right in there at the application level as well.]

**********NOAH'S COMMENTS STOP HERE -- I THINK IF WE CAN ITERATE TOWARD AGREEMENT IN THE AREAS RAISED ABOVE, THE REST WILL FOLLOW ************

As languages evolve, it becomes possible to speak of both backwards and forwards compatibility. We base our definitions of backwards and forwards compatibility on the [FOLDOC] definitions. There are two aspects of compatibility that are called out: software compatibility and schema compatibility. While it is often the case that they are directly related, sometimes they are not.

[Definition: A language change is backwards compatible if newer processors can process all instances of the old language. ] Backwards compatibility means that a newer version of a consumer can be rolled out in a way that does not break existing producers. A producer can send an older version of a message to a consumer that understands the new version and still have the message successfully processed. A software example is a word processor at version 5 being able to read and process version 4 documents. A schema example is a schema at version 5 being able to validate version 4 documents. This means that a producer can send an old version of a message to a consumer that understands the new version and still have the message successfully processed. In the case of Web services, this means that new Web services consumers, ones designed for the new version, will be able to process all instances of the old language.

[Definition: A language change is forwards compatible if older processors can process all instances of the newer language.] Forwards compatibility means that a newer version of a producer can be deployed in a way that does not break existing consumers. Of course the older consumer will not implement any new behavior, but a producer can send a newer version of an instance and still have the instance successfully processed. An example is a word processing software at version 4 being able to read and process version 5 documents. A schema example is a schema at version 4 being able to validate version 5 documents. This means that a producer can send a newer version of a message to an existing consumer and still have the message successfully processed. In the case of Web services, this means that existing Web service consumers, designed for a previous version of the language, will be able to process all instances of the new language.

In general, backwards compatibility means that existing documents can be used by updated consumers, and forwards compatibility means that newer documents can be used by existing consumers. Another way of thinking of this is in terms of message exchanges. Backwards compatibility is where the consumer is updated and forwards compatibility is where the producer is updated, as shown below:

Example 2: Evolution of Producers and/or Consumers

In broad terms, backwards compatibility means that newer producers can continue to use existing services, and forwards compatibility means that existing producers can use newer services

One form of compatibility is with respect to the strings, that is whether all the strings in one language are also strings in another language. Another form of compatibility is with respect to the information conveyed, that is whether the information conveyed by the strings in one language is conveyed by the same strings in another language. The strings could be compatible but the information conveyed is not compatible.

Compatibility is defined for the producer and consumer of an individual document instance. Most messaging specifications, such as Web Services, provide inputs and outputs. Using these definitions of compatibility, a Web service that updates its output message is considered a newer producer because it is sending a newer version of the message. Conversely, updating the input message makes the service a newer consumer because it is consuming a newer version of the message. All systems of inputs and outputs must consider both when making changes and determining compatibility. For full compatibility, any output messages changes must be forwards compatible (for the older receivers aka consumers) and any input message changes must be backwards compatible (for the older senders aka producers).

Compatibility can be defined in terms of the information and syntax sets. Any document that contains only V1 information set in the V1 syntax will be processable by V2, so V2 is backwards compatible with V1. Note the constraint that it is the V1 information set items in V1 syntax. If there was an extension to a language then that language is no longer V1. The compatibility guarantees are only for V1 and V2, not for other versions. More formally:

principle

Language V2 is backwards compatible with Language V1 if Language V2 Information set > (superset) Language V1 Information set.

Because V1 syntax set is larger than (superset of) V2 syntax, a V1 processor can process all V2 documents, so we say that V2 is forwards compatible with V1. More formally:

principle

Language V2 is forwards compatible with Language V1 if Language V1 Syntax > (superset) Language V2 Syntax.

These 2 principles can be combined together into:

principle

Language V2 is compatible with Language V1 if Language V1 Syntax > Language V2 Syntax > Language V2 Information Set > Language V1 Information set.

We have shown that forwards and backwards compatibility is only achievable through extensibility, and the process of compatible versioning is gradually narrowing the extensibility gap between the information set and the syntax set.

1.1.1.1 Partial Understanding

We have defined compatibility for all possible expressions of the language, that is full compatibility. There are many scenarios where a consumer will consume only part of the information set. Partial understanding affects both information and syntax set. Partial understanding usually results in a subset of the information set (because only part of the information set is understand). Interestingly, partial understanding usually increases or supersets the syntax. This is usually because the process of extracting the subset needed means that extra content, even those illegal under V1 syntax, is ignored.

We can imagine an application that only looks at given names and everything else is ignored. My favourite example of this is a _Baby Name_ Wizard. The application might use a simple XPath expression to extract the given name from inside the name. This is a different version of the Name Language, which we will call the Given Name Language. The syntax set for the Given Name Language is anything, given, anything. The information set for the Given Name language is given names. Because the Given Name Language syntax set is more relaxed that the Name Language V1, an addition of the middle name between the given and family is a compatible change for the Given Name Language.

Our principles with respect to compatibiity and language versioning need no change to deal with partial understanding.

principle

Partial Understanding a language is the creation of a language VPU such that Language VPU Syntax > Language V1 Syntax > Language V1 Information Set > Language VPU Information set.

Interestingly, partially understanding a language is creating a language VPU, such that the V1 language is a compatible change with the VPU. There may be many different versions that are all partial understandings of a language. We call these related languages "flavours" (note: dialects doesn't seem right, subsets also doesn't seem right). It may be very difficult for a language designer to know many different language flavours are in existance. However, a language designer can sometimes use the different flavours to their advantage in designing for a mixture of compatible and incompatible changes. Some changes could be compatible with some flavours but not other. It may be very useful to have some changes be compatible with some flavours, that is those consumers do not need to be updated or changed.

It is crucial to point out that the partially understood versions of the language are NOT producers of the language. They have relaxed the restrictions on the consuming side, but not on the production side of the language. If a flavour of a language was also used for consumption, it would have create an instance that is valid according to the Language V1 rules. Perhaps the only exceptions are if they are guaranteed that they will be producing for compatible flavours. Typically this is not the case and hard to determine, so the safest course is to produce according to the Language V1 rules.

We have shown how relaxing the constraints on a language when consuming instances of it can turn an otherwise incompatible change into a compatible change. We have also shown that abiding by the language constraints when producing instances is the safest course. Said more eloquently is the internet robustness principle, "be conservative in what you do, be liberal in what you accept from others" from [tcp].

This leads us to two versioning practices:

Good Practice

Use Least Partial Languages: Consumers should use a flavour of a language that has the least amount of understanding.

The least amount of understanding will be the most liberal or have the largest syntax set possible.

Good Practice

Produce no partial languages: Producuers should use the complete version of a language and no partial flavours of a language.

This guarantees interoperability with all the consumers and is the most conservative production possible.

EdNote: I think related to principle of least power. The least powerful the language, the easier to have partial understanding?

1.1.2 Open or Closed systems

The cost of changes that are not backward or forward compatible is often very high. All the software that uses the language must be updated to the newer version. The magnitude of that cost is directly related to whether the system in question is open or closed.

[Definition: A closed system is one in which all of the producers and consumers are more-or-less tightly connected and under the control of a single organization.] Closed systems can often provide integrity constraints across the entire system. A traditional database is a good example of a closed system: all of the database schemas are known at once, all of the tables are known to conform to the appropriate schema, and all of the elements in the each row are known to be valid for the schema to which the table conforms.

From a versioning perspective, it might be practical in a closed system to say that a new version of a particular language is being introduced into the system at such and such a time and all of the data that conforms to the previous version of the schema will be migrated to the new schema.

[Definition: An open system is one in which some producers and consumers are loosely connected or are not controlled by the same organization. The internet is a good example of an open system.]

In an open system, it's simply not practical to handle language evolution with universal, simultaneous, atomic upgrades to all of the affected software components. Existing producers and receivers outside the immediate control of the organization that has publishing a changed language will continue to use the previous version for some (possibly long) period of time.

Finally, it's important to remember that systems evolve over time and have different requirements at different stages in their life cycle. During development, when the first version of a language is under active development, it may be valuable to persue a much more aggressive, draconian versioning strategy. After a system is in production and there is an expectation of stability in the language, it may be necessary to proceed with more caution. Being prepared to move forward in a backwards and forwards compatible manner is the strongest argument for worrying about versioning at the very beginning of a project.

1.2 XML Terminology

There are many different systems for exchanging texts in languages, such as SQL, Java, XML, ECMAScript, C#. We will briefly describe some key refinements to our lexicon for XML. An XML language has a vocabulary that may use terms from one or more XML Namespaces (or none), each of which has a namespace name. [Definition: An XML language is an identifiable set of vocabulary terms with defined XML syntactic and semantic constraints. ] By XML language, we mean the set of elements and attributes, or instances, used by a particular application. The Name Language - consisting of name, given, family terms - has a namespace for the terms. We use the prefix "namens" to refer to that namespace. The Name Language could consist of terms from other vocabularies, such as Dublin Core or UBL. These terms each have their own namespaces, illustrating that a language can comprise vocabularies from multiple namespaces. An XML Namespace is a convenient container for collecting terms that are intended to be used together within a language or across languages. It provides a mechanism for creating globally unique names.

We shall use the term instance when speaking of sequences of characters (aka text) in XML. [Definition: An instance is a specific, discrete Text in XML format.] Documents are instances of a language. In XML, they must have a root element. A name document might have a name element as the root element. Alternatively, the name vocabulary may be used by a language such as purchase orders. The purchase order documents may contain name elements. Thus instances of a language are always part of a document and also may be the entire document. XML instances (and all other instances of markup languages) consist of markup and content. In the name example, the given and family elements including the end markers are the markup. The values between the start and end markers are the content. An instance has an information model. There are a variety of data models within and without the W3C, and the one standardized by the W3C is the XML infoset.

The XML related terms and their relationships are shown below

A stylesheet processor is a consumer of the XML document that it is processing (the producer isn't mentioned); in the Web services context the roles of producer and consumer alternate as messages are passed back and forth.Note that most Web service specifications provide definitions of inputs and outputs. By our definitions, a Web service that updates its output schema is considered a new producer. A service that updates its input schema is a new consumer.

1.3 Why Worry About Extensibility and Versioning?

As documents, or messages, are exchanged between applications, they are processed. Most applications are designed to discriminate between valid and invalid inputs. In order to have any sort of interoperability, a language must be defined or described in some normative way so that the terms "invalid" and "valid" have meaning.

There are a variety of tools that might be employed for this purpose (DTDs, W3C XML Schema, RELAX NG, Schematron, etc.). These tools might be augmented with normative prose documentation or even some application-specific validation logic. In many cases, the schema language is the only validation logic that is available.

It is almost unheard of for a single version of a language to be deployed without requiring some kind of augmentation. Invariably, the original language designer did not include certain terms and constraints. In fact, good designers should not try to define all the possible terms and constraints. This is sometimes called "boiling the ocean". Knowing that a language will not be all things to all people, a language designer can allow parties to extend instances of the language or the language itself. Typically the tools will allow the language designer to specify where extensions in the instance and extensions in the language are allowed. Of note, we do not call extending an instances of a language a new version. This limits our discussion of versioning to changes in a language, not changes to instances.

Whether you've deployed ten services, or a hundred, or a million, if you change a language in such a way that all those services will consider instances of the new language invalid, you've introduced a versioning problem with real costs.

Once a language is used outside of its development environment, there will be some cost associated with changing it: software, user expectations, and documentation may have to be updated to accomodate the change. Once a language is used in environments outside of a single realm of control, any changes made will introduce multiple versions of the language.

1.4 Why Do Languages Change?

There are many reasons why a different version of a language may be needed. A few of them include:

Bugs may need to be fixed. Production use may reveal defects or oversights that need to be fixed. This may involve changes to components of the language or changes to the semantics of existing components.
Changing requirements may motivate changes in the schema design. For example, a callback may be added to a service that performs some processing so that it is able to notify the caller when processing has completed.
Different flavors of a schema may be desirable. For example, the XHTML 1.0 Recommendation defines strict, transitional, and frameset schemas. All three of those schemas purport to define the same namespace, but they describe very different languages.

And additional schemas may be defined by other specifications, such as the XHTML Basic Recommendation.

Whatever the cause, over time, different versions of the language exist and designing applications to deal with this change in a predictable, useful way requires a versioning strategy.

1.5 Why Extend languages?

The primary motivation to allow instances of a language to be extended is to decentralize the task of designing, maintaining, and implementing extensions. It allows producers to change the instances without going through a centralized authority. It means that changes can occur at the producer or consumer without the language owner approving of them. Consider the effort that the HTML Working Group put into modularity of HTML. Without some decentralized process for extension, every single variant of HTML would have to be called something else or the HTML Working Group would have to agree to include it in the next revision of HTML.

1.6 How Do Languages Change?

At the most basic level, languages can change in only a few ways:

Content: The allowable content can evolve through addition or deletion. In XML, this becomes
- ElementsNew elements can be added, existing elements can be removed, or the acceptable number of occurences of an element can change. In addition, the content of an element could change from element only content to mixed content, or vice versa.
  
  For elements with simple content, the type or range of values that are acceptable can change.
- Attributes: New attributes can be added, existing attributes can be removed, or the type or range of values that are acceptable can change.
Semantics: The meaning of a an existing term can change.

Of course, the difference between two versions of a language can be an arbitrary number of these changes.

One of the most important aspects of a change is whether or not it is backwards or forwards compatible.

Some typical backwards- and forwards-compatible changes:

adding optional components (elements and/or attributes)
adding optional content, for example extending an enumeration

Some typical forwards-compatible changes:

Decreasing the maximum allowed number of occurrences of a component but not to less than the minimum
Decreasing the allowed range of a component

Some typical backwards-compatible changes:

Increasing the maximum allowed number of occurrences of a component
Increasing the allowed range of a component

Some typical incompatible changes:

changing the meaning or semantics of existing components
adding required components
removing required components
restricting a components content model, such as changing a choice to a sequence

1.7 Kinds of Languages

Ultimately, there are different kinds of languages. The versioning approaches and strategies that are appropriate for one kind of language may not be appropriate for another. Among the various kinds of vocabulares, we find:

Just Names: some languages don't actually identify elements or attributes, they're just lists of names. Using QNames to identify words in the WordNet database, for example, or the names of functions and operators in XPath2 are examples of "just name" languages.
Standalone: languages designed to be used more-or-less by themselves, for example XHTML, DocBook, or The TEI.
Containers: languages designed to be used as a wrapper or framework for some other language or payload, for example SOAP or WSDL.
Container Extensions: languages designed to extend or augment a particular class of container. Specifications that extend SOAP by defining SOAP header blocks, for example, to provide security, asynchrony or reliable messaging are examples of container extension languages.

There are a couple types of XML extension languages, element extension and attribute extension.
- Element Extension. Languages that are elements. SOAP, etc. are element extensions.
- Attribute or type Extensions. Langages that are types or attributes. These languages must exist in the context of an element. Sometimes called "parasite" languages as they require a "host" element. XLink is an example.
Mixtures: languages designed for, or often used for, encapsulating some semantics inside another language. For example, MathML might be mixed inside of another language.

This is by no means an exhaustive list. Nor are these categories completely clear cut. MathML can certainly be used standalone, for example, and languages like SVG are a combination of standalone, containers, and mixtures.

2 Language Questions

Given the definitions of compatibility described above, what questions must the language designer consider?

2.1 Can 3rd parties extend the language?

It is rarely desirable to prevent 3rd parties from extending languages, but it does happen. An example may be a tightly constrained security environment where distributed authoring is considered a _bug_ rather than a feature.

2.2 Can 3rd parties extend the language in a compatible way?

If so, a substitution mechanism is required for forwards compatibility. If an older consumer has no mechanism for dealing with new content, then forwards-compatible evolution isn't possible. One simple substitution mechanism is simply ignoring the unrecognized components.

2.3 Can 3rd parties extend the language in an incompatible way?

If so, and if compatible extensions are also possible, then it must be possible to identify incompatible changes so that they can override the substitution mechanism used for extensible changes.

In environments where unrecognized components are ignored, a _must understand_ component can be added to identify incompatible changes.

If compatible changes are not possible, then incompatible changes simply become the default. For example, WS-Security mandates that 3rd parties can only provide incompatible extensions. Unlike most languages, a security language has unique requirements where the consequences of ignored data can be severe. WS-Security accomplished this by specifying that all extensions are required to be understood and there is no substitution mechanism.

2.4 Can the designer extend the language in a compatible way?

As with 3rd parties compatible extensions, a substitution mechanism for the designer_s extensions is required for forwards compatibility.

2.5 Can the designer extend the language in an incompatible way?

In XML, the designer can always do this by using new namespace names, element names or version numbers. In other languages, this may not be possible because there is no mechanism for indicating the incompatible change.

2.6 Is the vocabulary a stand-alone language or an extension of another vocabulary?

A part of this question is whether the language depends on another language. That determines which, if any, facilities are provided for the containing language and what must be provided by a contained language.

SOAP is an example of a container language. The SOAP processing model applies uniformly to all headers, which may employ soap:mustUnderstand to identify incompatible changes, even though the contents of the SOAP headers are languages independent from SOAP.

2.7 What language form

Languages can be expressed in text, comma separted values, XML, SGML, binary, source code, and almost any kind of form. See the Architecture of the World-Wide Web section on data formats for more information - http://www.w3.org/TR/2004/REC-webarch-20041215/#formats.

2.8 What Schema language(s)?

Choosing a schema language or languages guides the language design in many ways. Some features, particularly extensibility, must be anticipated in the first version of a language in order to take advantage of the features of some schema languages.

In addition, various features may be incompatible across different languages. For example, writing a V2 compatible schema in W3C XML Schema requires special design, which is not required in a schema language such as RELAX NG. Some of the language design choices mandated by W3C XML Schema are discussed in other sections of this Finding.

2.9 Should extensions or versions be expressible in the Schema language?

The ability to write a schema for extensions or versions is directly affected by the schema design and the compatibility desires.

3 Language Decisions

Upon answering these questions, there are some key decisions that a language developer makes, whether they are consciously made or not.

3.1 Schema language design choices or constraints.

If the language can be extended in a compatible way, then a few specific schema design choices must be followed.

Wildcards are used to provide extensibility in XML Schema. If revisions to the Schema are to support substitution, specific schema designs must be used in conjunction with the wildcard. The main choices are: provide wildcards, provide Extension elements, or provide delimiter elements. Extension and delimiter elements are described in the new components in existing or new namespaces section. If Extension/delimiter elements are not provided, then a compatible V2 Schema cannot be written.

3.2 Substitution Mechanism.

Forwards compatibility can only be achieved by providing a substitution mechanism for Version 2 instances or Version 1 extensions to V1 without knowledge of V2. A V1 consumer must be able to transform any instances, such as V1 + extensions, to a V1 instance in order to process the instance. The _Must Ignore unknown_ rule is a simple substitution mechanism. This rule says that any extensions are _ignored_. Using it, a V1 + extensions document is transform into a V1 document by ignoring the extensions. Others substitution mechanisms exist, such as the fallback model in XSLT.

3.3 Component identification

The identification of components into language versions or extensions has a variety of general mechanisms related to namespaces. These are detailed in the Versioning section.

3.4 Identification of incompatible extensions

The identification of versions is covered by language identification, but 3rd parties cannot arbitrarily change versions or change namespaces. They may need a mechanism to indicate that an extension is an incompatible change. A couple of mechanisms are a _Must Understand_ identifier (such as a flag or list of required namespaces) or requiring that extensions are in substitution groups.

4 Identifying and Extending Languages

Designing extensibility into languages typically results in systems that are more loosely coupled. Extensibility allows authors to change instances without going through a centralized authority, and may allow the centralized authority greater opportunities for versioning. The common characteristic of a compatible change is the use of extensibility.

A supreme example of the benefits of extensibility is HTML. The first version of HTML was designed for extensibility; it said that _unknown markup_ may be encountered. An example of this in action is the addition of the IMG tag by the Mosaic browser team.

The first rule introduced in this Finding relating to extensibility is:

Good Practice

Allow Extensibility rule: Languages SHOULD be designed for extensibility.

A fundamental requirement for extensibility is to be able to determine the language of elements and attributes. XML Namespaces [13] provide a mechanism for associating a URI with an XML element or attribute name, thus specifying the language of the name. This also serves to prevent name collisions.

HTML did not have the ability to distinguish between the languages of extensions. This meant that authors could produce the same element name but with different interpretations, and software would have no way of determining which interpretation was applicable. This is a great part of the motivation to move from HTML to the XML vocabulary of HTML, XHTML.

W3C XML Schema [14] provides a mechanism called a wildcard, <xs:any>, for controlling where elements from certain namespaces are allowed. The wildcard indicates that elements in specified namespaces are allowed in instance documents where the wildcard occurs. This allows extension in a well-defined manner. A consumer of extended documents can identify and, depending upon its processing model, safely ignore the extensions it doesn't understand.

<xs:any> uses the namespace attribute to control what namespaces extension elements can come from. The most interesting values for this attribute are: ##any, which means one can extend the schema using an element from any possible namespace; ##other, which only allows extension elements from namespaces other than the target namespace of the schema; and ##targetnamespace, which only allows extension elements from the target namespace of the schema.

<xs:any> uses the processContents attribute to control how a XML parser validates extended elements. Permissible methods include _lax_ - validate any elements from supported namespaces but ignore all other elements, _strict__validate all elements, and _skip__validate no elements. This Finding recommends _lax_ validation, as it is the most flexible and is the typical choice for Web services specifications.

RDF/OWL and RelaxNG are 2 other popular technologies for schema design. They have different mechanisms for allowing and controlling schema evolution.

5 Example

Suppose that you have designed a language for handling personal information consisting of a single _Name_ element. The first version of the Name contains a _given_ and a _family_ element. There are a variety of strategies for extensibility and versioning, detailed later. This example will simply show the "new components in new namespace" strategy.

Example 3: New components in new namespace(s) schema Version 1

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.example.org/name/1" 
      xmlns:name="http://www.example.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="given" type="xs:string"/>
      <xs:element name="family" type="xs:string"/>
      <xs:element name="middle" type="xs:string" minOccurs="0"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

The language designer and 3rd parties can now use different namespaces for their versions. The language designer makes a variety of choices, particularly the schema language, that affect their strategy for namespaces. Thus the first rule for namespaces covers all language choices, but is open to extension based upon other design choices.First the rule for namespaces:

Good Practice

Allow Extensions in Other Namespace rule: The extensibility point SHOULD at least allow for extension in other namespaces.

The rule for allowing extensibility:

Good Practice

Full Extensibility rule: All XML Elements SHOULD allow for element extensibility after element definitions, and allow any attributes.

In general, an extension can be defined by a new specification that makes a normative reference to the earlier specification and then defines the new element. No permission should be needed from the authors of the specification to make such an extension. In fact, the major design point of XML namespaces is to allow decentralized extensions. The corollary is that permission is required for extensions in the same namespace. A namespace has an owner; non-owners changing the meaning of something can be harmful.

Attribute extensions do not have non-determinism issues because the attributes are always unordered and the model group for attributes uses a different mechanism for associating attributes with schema types than the model group for elements.

6 Understanding Extensions

Ideally, producers should be able to extend existing XML documents with new elements without consumers having to change existing implementations. Extensibility is one step towards this goal, but achieving compatibility also requires a processing model for the extensions. The behavior of software when it encounters an extension should be clear. For this, we introduce the next rule:

Good Practice

Provide Processing Model Rule: Languages SHOULD specify a processing model for dealing with extensions.

Achieving forwards-compatible evolution requires that the processing model must be a substitution mechanism. The instance containing the extension, which isn't known by the consumer, must be transformed into an instance which is of a type known by the consumer.

Good Practice

Provide Substitution model: Languages MUST provide a substitution model for forwards-compatible evolution.

The simplest substitution model that enables compatible changes is to ignore content that is not understood. This rule is:

Good Practice

Must Ignore Rule: Document consumers MUST ignore any XML attributes or elements in a valid XML document that they do not recognize.

This rule does not require that the elements be physically removed; only ignored for processing purposes. There is a great deal of historic usage of the Must Ignore rule. HTML 1, 2 and 3.2 follow the Must Ignore rule as they specify that any unknown start tags or end tags are mapped to nothing during tokenization. HTTP 1.1 [7] specifies that a consumer should ignore any headers it doesn't understand: "Unrecognized header fields SHOULD be ignored by the recipient and MUST be forwarded by transparent proxies." The Must Ignore rule for XML was first standardized in the WebDAV specification RFC 2518 [6] section 14 and later separately published as the Flexible XML Processing Profile [3].

There are two broad types of Must Ignore rules for dealing with extensions, either ignoring the entire tree or just the unknown element. The rule for ignoring the entire tree is:

Good Practice

Must Ignore All Rule: The Must Ignore rule applies to unrecognized elements and their descendents in data-oriented formats.

For example, if a message is received with unrecognized elements in a SOAP header block, they must be ignored unless marked as _Must Understand_ (see Rule 10 below). Note that this rule is not broken if the unrecognized elements are written to a log file. That is, _ignored_ doesn_t mean that unrecognized extensions can_t be processed; only that they can_t be the grounds for failure to process.

Other applications may need a different rule as the application will typically want to retain the content of an unknown element, perhaps for display purposes. The rule for ignoring the element only is:

Good Practice

Must Ignore Container Rule: The Must Ignore rule applies only to unrecognized elements in presentation-oriented formats.

This retains the element descendents in the processing model so that they can still affect interpretation of the document, such as for display purposes.

Ignoring content is a simple solution to the problem of substitution. In order to achieve a compatible evolution, the newer instances of a language must be transformable (or substitutable) into older instances. Object systems typically call this _polymorphism_, where a new type can behave as the old type.

Other substitution models have been successfully deployed. One such model is a fallback model, where alternate elements are provided if the consumer does not understand the extension. XSLT 2.0 provides such a model. Another model is that a transform from the new type to the old type is made available, either by value or reference.

As desirable as compatible evolution often is, sometimes a language may not want to allow it. In this model, a consumer will generate a fault if it finds a component it doesn_t understand. An example might be a security specification where a consumer must understand each and every extension. This suffers from the significant drawback that it does not allow compatible changes to occur in the language, as any changes require both consumer and producer to change.

7 Versioning

A language designer decides how new versions of their language, as well as extensions, are related to previous versions. They decide how to use namespace names, component names for their language, as well as possibly introducing versioning-specific components such as version identifiers and incompatible extension identifiers. When a new version of a language is required, the author must make a decision about the namespace name for names in the new language.

Version identification has traditionally been done with a decimal separating the major versions from the minor versions, ie _8.1_, _1.0_. Often the definition of a _major_ change is that it is incompatible, and the definition of a _minor_ change is that it is forwards- and/or backwards - compatible. Usually the first broadly available version starts at _1.0_. A compatible version change from 1.0 might be identified as _1.1_ and an incompatible change as _2.0_. It should be noted that this is idealistic as there abundant cases where this system does not hold. New major version identifiers are often aligned with product releases, or incompatible changes identified as a _minor_ change. A good example of an incompatible changed identified as a minor change is XML 1.1. XML 1.0 processors cannot process all XML 1.1 documents because XML 1.1 extended XML 1.0 where XML 1.0 does not allow such extension.

7.1 Versioning Strategies

Versioning is a broad and complex issue. Different communities have different notions about what constitutes a version, what constitutes a reasonable policy, and what the appropriate behavior is in the face of deviations from that policy. Historically, it has always proved more complicated in practice than in theory.

In broad terms, the approaches to versioning fall into a number of classes ranging from "none" to a "big bang":

None. No distinction is made between versions of the language. Applications are either expected not to care, or they are expected to cope with any version they encounter.
Compatible. Designers are expected to limit changes to those that are either backwards or forwards compatible, or both.
- Backwards compatibility. Applications are expected to behave properly if they receive an instance document of the "older" version of a language. Backwards compatible changes allow applications to behave properly if they receive an "older" version of the language.
- Forwards compatibility. Applications are expected to behave properly if they receive an instance document of the "newer" version of a language. Forwards compatible changes allow existing applications to behave properly if they receive a "newer" version of the language.
Flavors. Applications are expected to behave properly if they receive one of a set of flavors of the document type.
Big bang. Applications are expected to abort if they see an unexpected version.

There's no single approach that's always correct. Different application domains will choose different approaches. But by the same token, the approaches that are available depend on other choices, especially with respect to namespaces. This dependency makes it imperative to plan for versioning from the start. If you don't plan for versioning from the start, when you do decide to adopt a plan for versioning, you may be constrained in the available approaches by decisions that you've already made.

A language commonly goes through a lifecycle of iterative development followed by deployment followed by deployment of new versions. The point in the lifecycle will affect the selection of the versioning strategy for the language

Just as there are a number of approaches, there are a number of strategies for implementing an approach. The internet - including MIME, markup languages, and XML languages have succesfully used various strategies, either singly or in combination. Summaries of strategies and requirements have been produced for earlier technologies and guided XML Namespaces and Schema, such as [Web Architecture: Extensible Languages].

For any given approach, some strategies may be more appropriate than others. Among the strategies we find:

Must Understand. consumers must understand all of the elements and attributes received and are expected to abort processing if they do not. SOAP processors must understand headers that are explicitly identified to be mandatory.
Must Ignore Unknown. consumers must ignore elements or attributes that they do not understand. Sometimes the must understand and must ignore approaches can be combined for more selective use. SOAP processors must ignore headers they do not recognize unless the header explicitly identifies itself as one that must be understood.

There are 2 variations of the Must Ignore strategy:
- Must Ignore All This variation on must ignore requires the consumer to ignore an element or attribute it does not understand and, in the case of elements, all of the descendents of that element. Most data applications, such as Web services that use SOAP header blocks or WSDL extensions, adopt this approach to dealing with unexpected markup. For XML, the Must Ignore all rule was first standardized in the WebDAV specification RFC 2518 [WebDAV] section 14 and later separately published as the [FlexXMLP].
- Must Ignore Container. This variation on must ignore requires the consumer to ignore an element or attribute that it does not understand, but in the case of elements, to process the children of that element. The Must Ignore Container practice was described in [HTML 2.0]
Explicit Fallback. A language can provide mechanisms for explicit fallback if the extension is not supported. [MIME] provides multipart/alternative for equivalent, and hence fallback, representations of content. [HTML 4.0] uses this approach in the NOFRAMES element. In XML, the XML Inclusions specification [XInclude] provides a fallback element to handle the case where the putatively included resource cannot be retreived. There are many variations on where the fallback content can be found. For example, a schema language could specify that fallback content is found in an instance document, in a schema document, or even in the schema for the schema language.
Explicit Testing. A language can provide a mechanism for explicit testing. The XSLT Specification provides a conditional logic element and a function to test for the existence of extension functions. This allows designers of stylesheets to deal with different consumer capabilities in an explicit fashion.

Languages can choose a mixture of approaches. For example, XSLT provides both an explicit fallback mechanism for some conditions and explicit testing for others. The SOAP specification, another example, specifies Must Ignore as the default strategy and the ability to dynamically mark components as being in the Must Understand strategy.

7.2 Why Have a Strategy?

Different kinds of languages and different versioning strategies expose different problems. If you don't have a strategy at all, you are effectively choosing the "no versioning" strategy.

It's probably obvious that attempting to deploy a system that provides no versioning mechanism is frought with peril. Putting the burden of version "discovery" on consumers is probably impractical in anything except a closed system.

At the other end of the spectrum is the "big bang" approach which is also problematic.

"Big bang" is a very coarse-grained approach to versioning. It establishes a single version identifier, either a version number or namespace name, for an entire document.

The semantics of the "big bang" are that applications decide on the basis of the document version whether or not they know how to process that document. If the version isn't recognized, the entire document is rejected.. Typically, when introducing a new version using the big bang approach, all of the software that produces or consumes the instance documents is updated in a sweeping overhaul in which the entire system is brought down, the new software deployed and the system is restarted. This big bang approach to versioning is practical only in circumstances where there is a single controlling authority, and even in that case, it carries with it all manner of problems. The process can take a considerable amount of time, leaving the system out of commission for hours if not days. This can result in significant losses if the system is a key component of a revenue generating business process and the cost of coordinating the system overhaul can also be quite costly as well.

The "big bang" approach is appropriate when the new version is radically different from its predecessor. But in many cases, the changes are incremental and often a consumer could, in practice, cope with the new version. For example, it might be that there are many messages that don't use any features of the new version or perhaps it is appropriate to simply ignore elements that are not recognized.

For example, consider two services exchanging messages. Imagine that some future version of the language that they are using defines a new "priority" element. Because producers and consumers are distributed, it may happen that an old consumer, one unprepared for a priority element, encounters a message sent by a newer producer.

If big bang versioning is used, old systems will reject the new message. However, if the versioning strategy allowed the old consumer to simply ignore unrecognized content, it's quite possible that other components of the system could simply adapt to the previous behavior. In effect, the old system would ignore the priority element and its descendents so it would "see" a message that looks just like the old format it is expecting.

For the producer, the result would be that the request is fulfilled, though perhaps in a more or less timely fashion than expected. In many cases this may be better behavior than receiving an error. In particular, producers using the new format can be written to cope with the possibility that they will be speaking to old consumers.

If the new system needs to make sure that priority is respected, then it can change the purchase order's name or namespace to indicate that the new behavior is not considered backwards compatible.

What is needed is some sort of middle ground solution. An evolving system should be designed with backwards and forwards compatibility in mind.

One approach to mitigate against a complete overhaul is to run multiple versions of the system. One variant is offering both versions of the system, for example by using different URIs for the old and new resources. The request to one resource gets mapped to the other resource behind the scenes using a proxy or gateway. This "alternative" approach works when the intermediary can completely handle or generate the new information (for backwards compatibility) or ignore the new information (for forwards compatibility). For example, adding SSL security to a resource changes the URI but a Web server can typically handle mapping the https: URI to the older http: URI. If both URIs are maintained, then the addition is a compatible change. Another example is where new information is required, such as the priority, and the intermediary can apply a default value to provide the required priority. However, this too has its costs as multiple versions of the software must be supported and maintained over time and there is the added cost of developing the proxy or gateway between the two environments. Further, this does not work in scenarios where the intermediary cannot generate the new required content. For example, if a middle name is required in V2, a middle cannot be generated from just a family and a given name.

Having multiple versions naturally leads to identifying versions. One approach to version identification is to use version numbers, with a goal of using "major version" changes for incompatible changes and "minor version" changes for compatible ones. The version numbers can be contained in the instance documents, in the protocol messages containing in the instance document, or the address for the protocol messages (ie http://example.com/foo/v2).

Unfortunately, version numbers often wind up looking very similar to the big bang approach. In many approaches, each language is given a version identifier, almost always a number, that's incremented each time the language changes. Although it's possible to design a system with version numbers that enables both backward and forward compatibility - for example XSLT - typically a version change is treated as if that the new language is not backwards compatible with the old language.

Some efforts, such as HTTP, try to have the best of both worlds by allowing for extensibility (in HTTP's case, via headers) as well as version numbers that explicitly identify when a new version is backwards compatible with an old version.

One argument in favor of version numbers is that they allow one to determine what is a 'new version' and what is an 'old version'. But in practice this is not necessarily true. For example, RSS has 0.9x, 1.x, and 2.x versions, all being actively developed in parallel. In effect the version numbers, even though they appear to be ordered, are simply opaque identifiers. Using version numbers does not gaurantee that version 1+x has any particular relationship to version 1.

The self-describing and extensible nature of XML markup, and the addition of XML Namespaces, provide a much better framework for developing languages that can evolve.

8 Version Identification Strategies

There are a large variety of version identification designs. They range from many namespaces to only 1 namespace for all versions of a language. A few of the most common are listed below and described in more detail later.

all components in new namespace(s) for each version

ie version 1 consists of namespaces a + b, version 1.1 consists of namespaces c + d; or version 1 consists of namespace a, version 1.1 consists of namespace b.
all new components in new namespace(s) for each compatible version

ie version 1 consists of namespaces a + b; version 1.1 consists of namespaces a + b + c; version 2.0 consists of namespaces d + e.
all new components in existing or new namespace(s) for each compatible version

ie version 1 consists of namespace a, version 1.1 consists of namespace a, version 2 consists of namespace b; or version 1 consists of namespace a, version 1.1 consists of namespace a + b.
all new components in existing or new namespace(s) for each version and a version identifier

ie version 1 consists of namespace a + b + version attribute _1_, version 2 consists of namespace c + d + version attribute _2_.
all components in existing namespace(s) for each version (compatible and incompatible) and a version identifier

ie version 1 consists of namespace a + version attribute _1.0_, version 1.1 consists of namespace a + version attribute _1.1_, version 2.0 consists of namespace a + version attribute "2.0".

Whatever the design chosen, the language designer must decide the component name, namespace name, and any version identifier for new and all existing components. The trade-offs between the decisions relate to the importance of:

Supporting Compatible evolution.
namespaces for identifying compatible components. Changing namespace names is typically a very invasive change
A complete Schema for the language. We will see how some designs preclude full Schema description
Use of generic XML and namespace only (precluding vocabulary specific versions) tools. This itself is a trade-off because some generic XML tools (like XPath) are more difficult to use with multiple namespaces containing the same "thing", like XHTML's P element.

Elaborating on these designs is illustrative.

8.1 Version Strategy: all components in new namespace(s) for each version (#1)

The following names would be valid:

Example 4: All components in new namespace(s) instances

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/2">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/3">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/3/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/3">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2^nd example shows all the components in the same new namespace. The 3rd and 4^th example show an additional middle element in 2 different namespace names. The 3rd example comes from a namespace name that is in the same domain as the name element_s new namespace name. One reason for 2 namespaces is to modularize the language. The 4^th example shows a namespace name from a different domain for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

8.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)

In this strategy, the following names would be valid:

Example 5: New components in new namespace(s) instances

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2^nd and 3^rd example show an additional middle element in 2 different namespace names. The first middle, the 2^nd example, comes from a namespace name that is in the same domain as the name element_s namespace name. The 3^rd example shows a complete different namespace name for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

8.3 Version Strategy: all new components in new or existing namespace(s) for each compatible version (#3)

In this strategy, the following names would be valid:

Example 6: New components in new or existing namespace(s) instances

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2^nd example shows the use of the optional middle name in the name namespace. The 3^rd and 4^th example show an additional middle element in 2 different namespace names. The first middle, the 3rd example, comes from a namespace name that is in the same domain as the name element_s namespace name. The 4^th example shows a complete different namespace name for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

8.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)

Using a version identifier, the name instances would change to show the version of the name they use, such as:

Example 7: New components in existing or new namespace(s) with version identifier instances

<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/2" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

The last two examples show that the middle is now a mandatory part of the name. This is indicated by just the version number or a new namespace plus version number.

A significant downside with using version identifiers is that software that supports both versions of the name must perform special processing on top of XML and namespaces. For example, many components _bind_ XML types into particular programming language types. Custom software must process the version attribute before using any of the _binding_ software. In Web services, toolkits often take SOAP body content, parse it into types and invoke methods on the types. There are rarely _hooks_ for the custom code to intercept processing between the _SOAP_ processing and the _name_ processing. Further, if version attributes are used by any 3rd party extensions_say mid:middle has a version_then the schema cannot refer to the correct middle.

8.5 Version Strategy: all components in existing namespace(s) for each version and a version identifier(#5)

Using a version identifier, the name instances would change to show the version of the name they use, such as:

Example 8: New components in existing namespace(s) with version identifier instances

<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

The 2^nd example shows that the middle is an optional part of the name. The last example shows that the middle is a mandatory part of the name.

A downside with using new namespace names is that some tools, like XPath, can be harder to use in the face of new namespace names. Software that extracts the given and family name based upon the expanded name will often break if a new namespace name is used.

9 Indicating Incompatible changes

Given adoption of the Must Ignore rule, it is often the case that the creator of an extension or a new version wants to require that the consumer understand the extension, overriding the Must Ignore rule. The previous section showed how a version author could use new namespace names, element names, or version numbers to indicate an incompatible change. An extension author does not have these mechanisms available for indicating an incompatible or mandatory extension. A language provider that wants to allow extension authors to indicate incompatible extension must provide a mechanism for indicating that consumers must understand the extension.

Good Practice

Provide Must Understand Rule: Container languages SHOULD provide a _Must Understand_ model for indicating extensions that override a default Must Ignore Rule.

This rule and the Must Ignore rule work together to provide a stable and flexible processing model for extensions.

Must Understand flag

Arguably the simplest and most flexible over-ride technique is a Must Understand flag that indicates whether the item must be understood. The SOAP [8], WSDL [9], and WS-Policy [10] attributes and values for specifying understand are respectively: soap:mustUnderstand=_1_, wsdl:required=_1_, wsp:Usage=_wsp:Required_. SOAP is probably the most common case of a container that provides a Must Understand model. The default value is 0, which is effectively the Must Ignore rule.

A language designer can re-use an existing Must Understand model by constraining their language to an existing Must Understand model. A number of Web services specifications have done this by specifying that the components are SOAP header blocks, which explicitly brings in the SOAP Must Understand model.

A language designer can design a Must Understand model into their language. A Must Understand flag allows the producer to insert extensions into the container and use the Must Understand attribute to over-ride the must Ignore rule. This allows producers to extend instances without changing the extension element_s parent_s namespace, retaining backwards compatibility. Obviously the consumer must be extended to handle new extensions, but there is now a loose coupling between the language_s processing model and the extension_s processing model. A Must Understand flag is provided below:

Example 9: New components in new namespace(s) with Must Understand

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.example.org/name/1" 
      xmlns:name="http://www.example.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="given" type="xs:string"/>
      <xs:element name="family" type="xs:string"/>
      <xs:any namespace="##other" processContents="lax" 
            minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute ref="name:mustUnderstand"/>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:attribute name="mustUnderstand" type="xs:boolean"/>
</xs:schema>

An example of an instance of a 3rd party indicating that a middle component is an incompatible change:

Example 10: New components in existing or new namespace(s) instance with Must Understand

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1"
                name:mustUnderstand="true">
      Bryce
  </mid:middle>
</name>

Specification of a Must Understand flag must be treated carefully as it can be computationally expensive. Typically a processor will either: perform a scan for Must Understand components to ensure it can process the entire document, or incrementally process the instance and is prepared to rollback or undo any processing if an not understood Must Understand is found.

There are other refinements related to Must Understand. One example is providing an element that indicates which extension namespaces must be understood, which avoids the scan of the instance for Must Understand flags.

It is also possible to re-use the SOAP processing model with it's mustUnderstand.

Example 11: Using SOAP Must Understand

<soap:envelope>
  <soap:body>
    <name xmlns="http://www.example.org/name/1">
    <given>Dave</given>
    <family>Orchard</family>
   </name>
  </soap:body>
</soap:envelope>

<soap:envelope>
  <soap:header>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1"
                soap:mustUnderstand="true">
      Bryce
  </mid:middle>
  </soap:header>
  <soap:body>
    <name xmlns="http://www.example.org/name/1">
    <given>Dave</given>
    <family>Orchard</family>
   </name>
  </soap:body>
</soap:envelope>

9.1 Type changes

The various schema languages have a variety of defined mechanisms to indicate extensions that are incompatible. For example, XML Schema provides Type Extension and Substitution Groups. These mechanisms are described in Part 2.

10 Extension versus Versioning

The usage of namespace names for identifying components has led to the interesting situation where the distinction between an extension and a version can be quite blurred, depending upon the language designer_s choices.

One rough way of thinking of these two concepts is that extension is typically the addition of components over space; that is, designers other than the language_s creator are adding components. Versioning is typically the addition of components over time, under the designer_s explicit control. In either case, a change to the language may be done in a compatible or an incompatible way. The simple cases of extensions are compatible decentralized additions and versions are compatible or incompatible centralized changes are how we typically distinguish the terms. But these break down depending upon how the language is designed.

There are a couple of scenarios that illustrate the ambiguity in these terms. Imagine that version 1.0 of a Name consists of _First_ and _Last_ elements. A 3rd party author extends the Name with a _middle_ element in a new namespace which they control.

In scenario 1, the Name author decides to formally incorporate the middle name as an optional (and hence compatible) addition to the name, producing version 1.1 of the Name type. They do this by referring to the third party_s definition and namespace for middle names. This is typically considered a new _version_ of the Name and would probably result in a new schema definition. If the Name author re-uses namespace names for compatible revisions, there will be no difference in an instance document containing middle that is of Version 1.0 or Version 1.1 type. The instance documents are the same, and thus the distinction between a _version_ and an _extension_ is meaningless for an individual document.

In scenario 2, the middle author decides that the middle name is a mandatory part of the Name type. They were provided a mechanism for indicating an incompatible change and they use it. Now an instance of Name with the middle is incompatible with version 1.0 of the Name. What _version_ of the Name is this middle, and is the middle an _extension_ or a _version_? It isn_t 1.0. It_s probably more accurately thought of as a version defined by the 3rd party. Again, the presence of the _extension_ is actually an incompatible change.

These two examples_a 3rd party extension being added into a compatible version and a 3rd party extension resulting in an incompatible version_show the ability to specify (in)compatibility has blurred the distinction between these two terms.

11 Conclusion

This Finding is intended to motivate language designers to plan for versioning and extensibility in the languages from the very first version. It details the downsides of ignoring versioniong. To help the language designer provide versioning in their language, the finding describes a number of questions, decisions and rules for using XML, XML Namespaces and schema languages in XML language construction and extension. The main goal of the set of rules is to allow language designers to know their options for language design, and ideally make backwards- and forwards-compatible changes to their languages to achieve loose coupling between systems.

12 References

FOLDOC: Free Online Dictionary of Computing. (See http://wombat.doc.ic.ac.uk/foldoc/.)
FlexXMLP: Flexible XML Processing Profile. (See http://www.upnp.org/download/draft-goland-fxpp-01.txt.)
tcp: RFC 793, TCP (See http://www.ietf.org/rfc/rfc793.txt.)
MIME: RFC 1521, MIME. (See http://www.ietf.org/rfc/rfc1521.txt.)
HTML 2.0: RFC 1866, HTML 2.0. (See http://www.ietf.org/rfc/rfc1866.txt.)
WebDAV XMLIgnore post: Yaron GolandXML Ignore proposed for WebDAV (See http://lists.w3.org/Archives/Public/w3c-dist-auth/1997AprJun/0190.html.)
WebDAV: RFC 2518, WebDAV (See http://www.ietf.org/rfc/rfc2518.txt.)
HTML 4.0: HTML 4.0. (See http://www.w3.org/TR/1998/REC-html40-19980424/.)
TBL Mandatory Extensions: Berners-Lee. Web Architecture: Mandatory extensions. (See http://www.w3.org/DesignIssues/Mandatory.html.)
TBL Extensible languages: Berners-Lee. Web Architecture: Extensible languages. (See http://www.w3.org/DesignIssues/Extensible.html.)
TBL Evolution: Berners-Lee. Web Architecture: Evolvability. (See http://www.w3.org/DesignIssues/Evolution.html.)
Web Architecture: Extensible Languages: Berners-Lee and Connolly, ed. Web Architecture: Extensible Languages World Wide Web Consortium, 1998. (See http://www.w3.org/TR/1998/NOTE-webarch-extlang-19980210.)
HTML Document types: Connolly, ed. HTML Document dialects World Wide Web Consortium, 1996. (See http://www.w3.org/MarkUp/WD-doctypes.)
SOAP 1.2: W3C Recommendation, SOAP 1.2 Part 1: Messaging Framework (See http://www.w3.org/TR/SOAP/.)
WSDL 1.1: W3C Note, WSDL 1.1 (See http://www.w3.org/TR/WSDL/.)
XML 1.0: W3C Recommendation, XML 1.0 (See http://www.w3.org/TR/REC-xml.)
XInclude: W3C Working Draft, XML Inclusions (See http://www.w3.org/TR-Xinclude.)
XML Namespaces: W3C Recommendation, XML Namespaces (See http://www.w3.org/TR/REC-xml-names.)
XML Schema Part 2: W3C Recommendation, XML Schema, Part 2 (See http://www.w3.org/TR/xmlschema-2.)
XML Schema Wildcard Test Collection: XML Schema Wildcard Test collection (See http://www.w3.org/XML/2001/05/xmlschema-test-collection/result-ms-wildcards.htm.)
XFront Schema Best Practices: XFront Schema Best Practices (See http://www.xfront.com/BestPracticesHomepage.html.)
XML.com Schema Design Patterns: Dare ObasanjoXML.com Schema design patterns (See http://www.xml.com/pub/a/2002/07/03/schema_design.html.)
Dave Orchard writings on Extensibility and Versioning: Dave Orchard writings on extensibility and versioning (See http://www.pacificspirit.com/Authoring/Compatibility.)

13 Acknowledgements

The author thanks the many reviewers that have contributed to the article, particularly David Bau, William Cox, Ed Dumbill, Chris Ferris, Yaron Goland, Hal Lockhart, Mark Nottingham, Jeffrey Schlimmer, Cliff Schmidt, and Norman Walsh.

[Editorial Draft] Extending and Versioning Languages Part 1

Draft TAG Finding 26 July 2006 With Comments by Noah Mendelsohn added 27 August 2006

Abstract

Status of this Document

Table of Contents

0 Overall Comments from Noah

1 Introduction

1.1 Terminology

1.1.1 Compatibility

1.1.1.1 Partial Understanding

1.1.2 Open or Closed systems

1.2 XML Terminology

1.3 Why Worry About Extensibility and Versioning?

1.4 Why Do Languages Change?

1.5 Why Extend languages?

1.6 How Do Languages Change?

1.7 Kinds of Languages

2 Language Questions

2.1 Can 3rd parties extend the language?

2.2 Can 3rd parties extend the language in a compatible way?

2.3 Can 3rd parties extend the language in an incompatible way?

2.4 Can the designer extend the language in a compatible way?

2.5 Can the designer extend the language in an incompatible way?

2.6 Is the vocabulary a stand-alone language or an extension of another vocabulary?

2.7 What language form

2.8 What Schema language(s)?

2.9 Should extensions or versions be expressible in the Schema language?

3 Language Decisions

3.1 Schema language design choices or constraints.

3.2 Substitution Mechanism.

3.3 Component identification

3.4 Identification of incompatible extensions

4 Identifying and Extending Languages

5 Example

6 Understanding Extensions

7 Versioning

7.1 Versioning Strategies

7.2 Why Have a Strategy?

8 Version Identification Strategies

8.1 Version Strategy: all components in new namespace(s) for each version (#1)

8.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)

8.3 Version Strategy: all new components in new or existing namespace(s) for each compatible version (#3)

8.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)

8.5 Version Strategy: all components in existing namespace(s) for each version and a version identifier(#5)

9 Indicating Incompatible changes

9.1 Type changes

10 Extension versus Versioning

11 Conclusion

12 References

13 Acknowledgements