W3C
National Institute of Standards and Technology

XML:id Conformance Test Suites Process Document

January 2005

This version:
Latest version:
Editor:
Sandra I. Martinez, NIST Working Group representative

Status

This is the January 2005 XML:id Conformance Test Suites Process Document.

Comments on this document are invited and are to be sent to the XML:id public mailing list public-xml-id.

This document has been produced as part of the W3C XML Activity. The authors of this document are the XML WG members.

Table of contents

1. Introduction

A joined effort between W3C and NIST has been established to produce a conformance test suite (XML:id TS) for the XML:id recommendation. NIST has agreed to allocate resources to update and maintain the current test suite. Also, the test suites will be jointly developed by these two parties and will take the form of a public framework as explained below.

2. Requirements

  1. The XML:id test suites should be developed in a public framework. The XML WG welcomes the participation of interested parties in developing the test suites.
  2. The XML:id test suites are intended to be used as a tool to aid implementors in developing XML:id implementations. Validation and certification of these implementations are outside the scope of this work. The tests and test suites will be provided for information and help only. However, we intend to produce as comprehensive, functional and general tests as possible, and this should be the overall goal in designing and implementing the XML:id TS.
  3. The test suites could be hosted on the W3C site once developed. Where the test suite should reside while being developed is still being discussed. They could also after development be mirrored at various sites in order to simplify access and enhance availability to the community.

3. Parties involved

The XML:id TS will be produced in a public framework with contributions from developers and companies in the community. The XML:id TS will be coordinated and supervised by the XML WG and NIST.

4. Procedural Issues

There is a need to have a stable mechanism for submitting tests to the test suite. The points below assume that there is a technically stable mechanism for submitting tests, saving information about those tests, version handling and so forth.

It is presupposed that developers or organizations submitting tests have done a sanity check with regard to the functionality of the tests by referring to the relevant specification.

Communication

The main channel of communication for the XML:id TS will be the XML:id public mailing list public-xml-id.

Submitting tests (or whole test suites)

The main channel of communication for submitting tests to the XML:id TS will be the XML:id public mailing list public-xml-id. An archive is available at http://lists.w3.org/Archives/Public/www-xml-XML:id-comments/.

In order to simplify for individuals and companies the production of the test suite and, if they so wish, contribute with tests of their own, we propose the following process for submitting tests:

  1. The test, or series of tests, get submitted to the framework. Submitter should also send a notification to the mailing list that will be set up for the XML:id TS to indicate that he/she has submitted a test to the XML:id TS framework. Submitting parties are also encouraged to check the tests they are submitting against the list of tests needed.
  2. It is acknowledged (presumably by the moderators/reviewers of the test suite) that the test has been received. This should be done via email directly to the submitter, and a copy should be sent to the mailing list. The test receives the "received" status.
  3. The test together with the necessary documentation is saved for future reference.

Receiving and reviewing tests

The test suite's moderators investigate the test given the following criteria. At each point is also indicated possible reasons for not accepting to publish the particular test.

  1. Relevance for the particular implementation of XML:id. It can here be decided that the particular test is irrelevant to the TS since it tests a feature that is not in the specification.
  2. Part of the specification being tested. If there are other, previously submitted tests that are functionally equivalent and test the same part of the specification, this test should either be kept for archive purposes but not published if it is functionally equivalent to a previously submitted test, or take the previously submitted test's place in the test suite.
  3. Quality of code and documentation (the code should be stable and not contain any errors, the documentation should be written in accordance with the XML file in which specification, functionality and possible scenarios are documented).
  4. Scenarios that underly the particular test layout and its intended scope.
  5. Development guide for the XInlcude TS, once developed. The existence of these guidelines will be communicated through the XML:id TS mailing list.

If the test is decided to be stable, which means that it is relevant, that it indeed tests a particular identifiable part of the specification, and that it is not wrongly written, it becomes "Reviewed and Stable" and receives this status.

Tests that at this or some other stage are judged not to be appropriate for publishing receive the "Inappropriate" status. These tests should be kept for archive purposes.

Reevaluating tests

In cases where tests have received the "Reviewed and Stable" status but are found to be erroneous, the following procedure is proposed:

If it is found that a test that has been called "Reviewed and Stable" actually is not stable (if the tests are not correctly written or for any other reason), this particular test gets submitted to the XML WG for further consideration. Possible outcomes of this stage is that the test, after consideration, is judged to be

  1. Correct, in which case it goes back to "Reviewed and Stable" status
  2. In need of further investigation and possibly rewriting (after which it receives the "Reviewed and Stable" status)
  3. Inappropriate for publication according to the previous point.
  4. If a test is judged to be correct after consideration, it keeps the "Reviewed and Stable" status.
  5. If a test as a result of this re-evaluation is judged to be inappropriate, it receives the "Inappropriate" status.

Documentation

We propose that each test or whole suite comes fully documented with regard to the following (there will be an XInlcude wrapper for this kind of information available from the TS website):

  1. Name of submitter, date of creation, and, if appropriate, company.
  2. Suggested ID for the test (this may be changed by the moderator to comply with the identification mechanism of the technical platform).
  3. Part of specification being tested. Areas in the specification being tested should be suitably indicated, either by giving a link to the relevant part of the specification, or by indicating, in text, that it is this part of the specification that is being tested.
  4. If appropriate, the scenario which has preceded the writing of the test. This could take the form of a textual description, eg. "This test investigates whether a user of XX can use a feature such that so-and-so happens on mouse over".

Status of the test suite according to the above

  1. The XML:id Test Suites should consist only of submitted, reviewed and stable tests.
  2. The XML:id TS will also consist of documentation on the tests it contains, scenarios that have served as a base for them, commented code and possibly comments from the editors.

It is proposed that the W3C XML WG representative act as moderator and controller for incoming tests. The XML:id TS moderator is choeon by the XML WG. All tests should be kept for archive purposes, whether they get published or not.

5. Conformance

The XML:id TS aims, as explained above, at helping implementors to write applications that support the XML:id recommendation. In no way are these conformance tests in the sense of providing companies or institutions with certification of XML:id support. The only claim that could be made is that a particular implementation is conformant to a particular version of the XML:id TS. There are two cases of results of running the test suite:

  1. The implementation fails to pass the test suite. In this case it can be asserted that the implementation fails to meet the relevant XML:id recommendation.
  2. The implementation passes the test suite. In this case all that can be asserted is that the implementation is conformant to that particular version of the XML:id TS.

6. Ownership

The XML:id TS will use the following grant of license:

Grant of License for Contributions under the W3C Software License

The Contributor hereby grants to the W3C, a perpetual, nonexclusive, royalty-free, world-wide right and license under any Contributor copyrights in this contribution to copy, publish and distribute the contribution under the W3C Software License (19980720), as well as a right and license of the same scope to any derivative works prepared by the W3C and based on, or incorporating all or part of the contribution.

The Contributor vouches that she/he has all rights necessary to contribute the Materials in a way that does not violate copyright, patent, and trademark rights; contractual obligations, or libel regulations.

Contributor further agrees that any derivative works of this contribution prepared by the W3C shalll be solely owned by the W3C. The Contributor agrees that all contributed Materials when published or otherwise distributed by the W3C will be governed by the W3C Software License (19980720).

W3C will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly signalled in the distributed version thereof. The W3C makes no a-priori commitment to support or distribute contributions.

7. Timeframe

Preliminary dates: