Copyright © 2002 W3C® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use, and software licensing rules apply.
This document outlines the process for collecting and maintaining test cases for an XForms 1.0 test suite consisting of "Basic Effectivity" tests, in accordance with W3C QA procedures.
Last Update: $Date: 2002/08/19 16:47:21 $
This is the @@@date XForms Test Suites Process Document.
Comments on this document are invited and are to be sent to the XForms public mailing list www-forms@w3.org. An archive is available.
1 Introduction
2 Requirements
3 Parties Involved
4 Procedural Issues
4.1 Communication
4.2 Submitting Tests
4.3 Receiving and reviewing tests
4.4 Reevaluating Tests
4.5 Documentation
5 Conformance
6 Ownership
7 Timeframe
The XForms Working Group and the community in general agree upon the need to produce test suite for XForms 1.0. The test suite will be jointly developed by interested parties and will take the form of a public framework as explained below.
The XForms test suite must be developed in a public framework. The XForms WG welcomes the participation of interested parties in developing the test suites.
The XForms test suite is intended to be used as a tool to aid implementors in developing XForms implementations. Validation and certification of these implementations are outside the scope of this work. The tests and test suites will be provided for information and help only.
The test suites could be hosted on the W3C site once developed. Where the test suite should reside while being developed is still being discussed. They could also after development be mirrored at various sites in order to simplify access and enhance availability to the community.
The test suite will be produced in a public framework with contributions from developers and companies in the community. The test suite will be coordinated and supervised by the XForms Working Group. The representatives will use resources from their organizations as well as invite individuals and companies (through representatives) to allow for a larger number of people to get involved.
Since this will be a public framework, there are a series of procedural issues that need to be resolved. There is a need to have a stable mechanism for submitting tests to the test suite. The test suite editor (QA Moderator) will maintain a list of tests needed that will be available at the XForms test suite development web site.
The main channel of communication for the test suite will be the XForms public mailing list www-forms@w3.org.
In order to simplify for individuals and companies the production of the test suite and, if they so wish, contribute with tests of their own, we propose the following process for submitting tests:
The test, or series of tests, are to be submitted to the mailing list. The submitter must agree to the IPR terms given at 6 Ownership. Submitting parties are encouraged to check the tests they are submitting against the list of tests needed.
The test(s) will be reviewed according to the following section.
The test suite's moderators investigate the test given the following criteria. At each point is also indicated possible reasons for not accepting to publish the particular test.
Relevance. The moderators can here be decide that the particular test is irrelevant to the test suite, or modify the test to make it relevant.
Part of the specification being tested. If there are existing tests that are functionally equivalent, the moderators can decide that the test should be rejected, or kept for archive purposes, or added to the test suite (replacing the previously submitted test), or adapted to cover a new area of the specification.
Quality of code and documentation. The moderators may reject tests that are not stable, or contain errors, or other problems that would make the test difficult to use or adapt.
If the test is decided to be stable, which means that it is relevant, that it indeed tests a particular identifiable part of the specification, and that it is not wrongly written, it becomes "Reviewed and Stable" and receives this status.
Tests that at this or some other stage are judged not to be appropriate for publishing receive the "Inappropriate" status. These tests should be kept for archive purposes.
The XForms Working Group is the final arbiter in the event of any disputes.
If it is found that a test that has been called "Reviewed and Stable" actually is not stable (if the tests are not correctly written or for any other reason), this particular test gets submitted to the XForms Working Group for further consideration. Possible outcomes of this stage is that the test, after consideration, are:
Correct, in which case it goes back to "Reviewed and Stable" status.
In need of further investigation and possibly rewriting (after which it receives the "Reviewed and Stable" status).
Inappropriate, in which case it goes to the "Inappropriate" status.
Each test should be fully documented with the following information:
Name of submitter, date of creation, and, if appropriate, company.
Suggested ID for the test (which may be modified by the moderator).
Part of specification being tested.
If appropriate, the scenario which has preceded the writing of the test.
The XForms test suite aims, as explained above, to help implementors to write applications that support the XForms specification. These tests do not provide companies or institutions with certification of DOM support. The only claim that could be made is that a particular implementation is conformant to a particular version of the test suite. There are two cases of results of running the test suite:
Failing one or more of the tests asserts that the implementation fails to conform to the specification.
Passing all of the tests does not guarantee full compliance of an implementation to the specification.
All contributions to the XForms test suite must be under the terms of the W3C Document License. In summary this states that:
Anyone can copy, but not modify the test suite.
Copies must provide a link back to the original test sute.
W3C will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly signalled in the distributed version thereof. The W3C makes no a-priori commitment to support or distribute contributions.