Copyright ©2001 W3C® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This document defines a common framework for building conformance test suites and tools for W3C specifications. It presents organizational, operational, and procedural guidelines for groups undertaking conformance materials development, as well as technical guidelines for the test suites (TS) and test tools themselves.
This document is a Note, made available by the W3C Quality Assurance Activity (QA) for discussion at the first QA face-to-face meeting.
This version is a first, incomplete draft. Some sections are written, others are only outlined. Some outlines could be considered fairly complete, but others -- particularly those in section 4 -- are not. Open issues and other incompletions are flagged with "@@". In different sections, for illustrative purposes, the style varies from prose, to Guidelines, to Guidelines-plus-Checkpoints, to the latter plus proposed external references to "Examples & Techniques". In this document version, guidelines and checkpoints are not yet numbered or formatted, and the proposed "Techniques" external references are in fact illustrated by internal blocks of text (not external content).
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
Please send comments to www-qa@w3.org.
1. Document goals and scope
1.1 Goals
1.2 Scope
1.3 Applicable domain
1.4 Document orientation & structure
1.5 Level & target audience
1.6 Terminology
2. Organizational considerations
2.1 W3C process requirements
2.2 De facto QA process conventions
2.3 Where to do TS development
2.3.1 Test materials home
2.3.2 Where to build the suite
3. Procedural & operational framework
3.1 Principal aims of TS development
3.2 WG Charter considerations
3.3 In-WG process
3.3.1 Starting & synchronizing with spec progression
3.3.2 First, Create a QA (Test) Process Document
3.3.3 Define and setup logistics
3.3.4 Who (staffing roles)
3.3.5 Build requirements
3.3.6 Contribution requirements
3.3.7 Repository, version, and access control
3.3.8 IPR and copyright
3.3.9 Distribution & access considerations
3.3.10 Accessibility & Internationalization
requirements/harmonization
3.3.11 Maintenance procedures
3.4 WG relationship to QA Activity
3.5 Transferring a TS from outside W3C
3.6 Logos & branding
3.7 other Ops&Proc topics
4. Technical Framework
4.1 Caveat about taxonomy dependencies
4.2 Idealized testing methodology (in 4-5 steps)
4.3 Idealized TS-build methodology (in 4-5 steps)
4.4 TS components (taxonomy-sensitive)
4.5 TR/TA identification or extraction
4.6 TP synthesis or derivation
4.7 Principles of Test Suite design & content
4.8 Principles of Test Case design & content
4.9 Serialization/versioning
4.10 Results and reporting
4.11 User documentation
4.12 Other technical topics
5. Conformance
6.
Acknowledgments
7. References
This document defines a common framework for building conformance test materials for W3C Recommendations. It presents organizational, operational, and procedural guidelines for groups undertaking conformance materials development, as well as technical guidelines for the test suites (TS) and tools themselves.
Foremost amongst the purposes of defining a common framework is the principal goal of QA development itself:
More specific goals of this framework document include:
Ideally, this framework document would provide to those undertaking test suite projects,
although the degree to which a simple cookbook guide is achievable is complicated by the diversity of potential test suites and tools, as well as by organizational diversity.
As a W3C Note, this document aims to serve as a vehicle and framework for discussion within the W3C Quality Assurance (QA) activity, in the fora of the QA Working Group [QAWG] and QA Interest Group [QAIG].
This does not preclude that portions of these guidelines may end up with a status other than Note, either by incorporation into the W3C Process Document [PROCESS], or by processing in the W3C Recommendation track.
The scope of this document encompasses:
Outside the scope of this document are:
The QA Activity considers W3C specification guidelines -- on topics including clear conformance clauses, enumeration of discretionary behaviors, and identification and addressability of testable assertions -- to be a critically important part of the QA programme of work. They will be addressed in a separate document [@@ref, when there is one].
Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:
This framework and its guidelines should be applicable and useful throughout this diverse context.
This is not intended to be a survey of existing W3C practice. The QA matrix [MATRIX] is a survey of all known existing and in-progress conformance test materials for W3C standards. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.
As much as possible, the framework will be presented as a set of informative guidelines for WG members and TS developers. Some of the WAI standards, such as WCAG [WCAG10], provide a good model of such a guideline-oriented style, using verifiable checkpoints
To deal with the complexities noted in the preceding section, the guidelines of this document will as much as possible be stated with general, cross-taxonomy applicability. A companion document, "QA Framework Examples and Techniques" [@@EX-TECH], will have additional detailed content linked from some of these individual guidelines. The linked material will explore, define, give examples of taxonomy-dependent details, as well as present techniques for implementing guidelines.
Although these are referred to as "informative guidelines", it is recognized that some of this content may one day become normative -- one of the meta-standards for development of W3C standards.
This framework specification aims to be comprehensive -- it spans the breadth of organizational, process, and technical topics which test suite developers are likely to encounter. While it should be useful across a range of reader experience levels, it is an explicit goal to be useful and accessible to readers at the start of QA requirements planning or development of test suite materials.
Languages for test case definition (TCDL), test results expression (e.g., EARL), are an important part of a conformance test framework. In this specification, they will be briefly mentioned in context. However their examination in any detail will be reserved for later specifications -- Advanced Topics.
The principal intended audience is the members of new and existing Working Groups (WGs), especially those who have interest in or will be involved in the QA activities of their WG. These will include test-suite builders, managers/editors, and test case contributors.
One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in this framework specification are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.
(@@Ed. note -- following is not uniformly implemented yet.)
Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.
This section is informative background -- there are no guidelines in here.
The current W3C Process Document says virtually nothing about conformance testing. Two mentions may be found:
No other process documents yet address the requirements or processes of building test suites and tools. One identified deliverable of this QA activity may be changes to the W3C Process Document, or production of other normative, QA-specific process documents.
Nevertheless, there is a body of contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and validators, in various states of development.
A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:
Therefore, based on contemporary practice and coordination with related activities, a sensible process requirement is:
Guideline. Working Groups should consider completion and publication of significant test materials to be a criterion for CR-exit and PR-entrance.
Guideline. Test suites and tools should ultimately reside in W3C.
Checkpoint. W3C should provide secure and reliable repository location for the test materials.
Checkpoint. W3C should have ultimate responsibility -- though not necessarily sole responsibility -- for the ongoing evolution of test suites and tools, after some milestone of operational maturity of the materials.
Explanation & Rationale. This does not necessarily mean that the WG must solely build the test suites and tools -- in fact there are good reasons why external parties should have a strong participation during the building of the materials (see @@below). But once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), they should reside in and be the responsibility of W3C, and in particular the Working Group responsible for the subject standard. Reasons:
If the WG responsible for a standard becomes defunct (@@does this ever happen, for a REC that still stands?), then another home for the test materials would be found within W3C. Those responsible for ongoing interpretation and maintenance (errata) of the standard should be designated.
Existing test materials developments show several patterns:
The determining factors for a new test suite effort will be:
Building test suites is expensive. See below. (@@link to end of "WG Charter...", its EX-TECH reference, or even directly into that EX-TECH reference from here.)
The decision might not always be W3C's alone. In several past cases, an outside group has taken the initiative and begun work. Example: XSLT test suite and tools [XSLT-TEST] are being built by the OASIS XSLT/Xpath Conformance TC.
Guideline. The TS development process benefits from the involvement of both WG-member experts on the subject standard, and neutral participants who are uninvolved in writing the standard.
Explanation & Rationale. Regardless of the development pattern -- which organizations do the development work -- experience has shown that there is high value in having active participation by parties other than the people who wrote or are writing the REC. The general benefits of test suite work are:
These benefits pertain, to some degree, regardless of where the work is done. However, in a QA mail thread [EXTERN-TA] it is asserted, and there is apparent consensus based on test suite experiences, that the participation of third parties helps to detect some kinds of spec-testability problems that may be invisible to the writers of the subject standard.
Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.
The over-arching goal is stated in the QA Activity Statement:
More specific goals of test materials development should include:
The goals of a common W3C QA framework do not encompass:
While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, test suites and materials development should not arbitrarily preclude:
Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.
Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any braning-related goals are articulated.
(@@Ed. note: We're trying this section in the style of WAI Guidelines/Checkpoints.)
Guideline. Working Group charters should address goals and plans for test suites and tools.
Explanation & Rationale. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.
Checkpoint. New WG charters SHALL address test suite plans.
Checkpoint. Existing WG charters SHOULD (MAY?) be amended with test suite plans.
Checkpoint. Charters should express commitment to general intent and plans.
Explanation. Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK]. A range of possible commitments has been identified. See "Examples & Techniques" [@@EX-TECH].
[...in EX-TECH, we would have:
The range of possibilities for WG commitment is:
0.) WG makes no firm commitment to address quality (not an option). 1.) WG promises to think about testability when writing their documents. 2.) WG makes an effort to have QA expertise among their membership. 3.) Beyond a few normative examples, WG aims to have numerous normative cases in the body of the Rec. 4.) WG commits that a test suite, not necessarily complete and thorough, will exist somewhere before they go to Rec. 5.) WG commits that they review and fully approve a test suite, not necessarily complete and thorough, before they go to Rec. 6.) WG commits to reviewing test cases on a continuing basis, including after going to Rec. 7.) WG will attempt to generate a complete catalog of test cases before going to Rec. 8.) WG insists on a complete catalog and partial implementation of a test suite before going to Rec. 9.) WG insists on a test suite, not necessarily complete and thorough, and provides an official test harness, before going to Rec. 10.) WG commits to writing test cases themselves and delivers a test harness and suite at some point. 11.) WG insists on a complete test suite before going to Rec.
This 12-point enumeration is derived from proposal to the QA mail list, after the 4/2001 QA Workshop. Point 0 is not an option -- WGs are required to commit to address quality at some level of effort.
The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.
/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]
Checkpoint. The WG charter should address where and how conformance test materials will be produced.
Explanation. There are several possibilities for producing conformance test materials:
Checkpoint. The WG charter TS plans should address staffing commitments.
Explanation. There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].
[...in EX-TECH, we would have:
-- quick tabular summary of size-of-effort data, from section 2.3 of http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.
-- quantify some examples, e.g., estimated XML uptake effort?
/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]
- (charter redux: outlines and commits to the WG plan and intentions at general level)
- start: the sooner, the better
- early benefits of, e.g., TA analysis
- when to start TS work (WD, CR, ...)
- coordination with spec (REC) progression
- WG produce a QA (Test) Process Document addressing...
(bullet list here, each to be expanded in following sections)
- examples: DOM; SVG (virtually a process doc: testManual + cvsManual),
- (distill set of guidelines and checkpoints from the SVG & DOM examples).
- mail lists (make a -ts list),
- web page intro (../Test); examples: SVG, DOM.
- ...etc...
- manager/moderator/editor (3 terms for same thing)
- some WG staff (and other W3C members)
- some QA staff?
- some 3rd party (e.g., for TR analysis)?
- content goals (BE, DT, numbers, etc)
- decide and document (technical) process & methodology
- define review/acceptance procedures and criteria
- define Submission Requirements
- define review & acceptance criteria (probably same as above)
- plan (identify) tools, methods, & infrastructure for accepting and integrating
- extract guidelines from SVG (look also at DOM TS Process).
- on contributed materials
- on the TS itself
- packaging
- procedures
- public & user documentation
- WCAG requirements for content ("A" minimal guideline)
- ATAG or UAAG requirements for tools
- I18n considerations?
- pubrules and documents
- spec: errata (Process Doc? Practical?)
- TS correctness challenges
- resolving
- fixing
- appeals (principles/guidelines from email threads)
The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.
Potential relationships between Working Groups and the QA Activity include:
Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.
- parties must to sort out IPR
- charter amendment? yes.
- need to sort out new home (WG), staffing, and other issues at "Charter" level of detail.
(several guidelines and checkpoints can be distilled from XML transfer example)
- after the transfer, all of the above, "In-WG", stuff pertains.
- WAI does this for content; CSS also and basic HTML validation.
- Controversial topic, as email thread and UAAG CR comments show.
- Taxonomy dependent: content (WCAG, CSS, etc) is less difficult/controversial; UAs will be contentious.
- ...any?...
(Right, this is a real hodge-podge still. Very little development yet, as I've focused on previous sections. Main issues will be similar -- prosey vs. guidelines? separate document for details, like WAI "Techniques"? Taxonomy dependencies will be more acute here, as illustrated by: HTML validator; DOM TS; SVG TS)
- for content&methodology
- will impose per-family splits in some of the following technical guidelines
- try to articulate a guideline at a general level
- handle details with companion document, "Examples & STechniques".
- several potential views
- tester activity view: configure ... test ... judge ... record ... report
- (or) data view: TCDL ... TC ... { Infosetized XML | EARL }
- (or) ??? view: (what view is most useful?).
- (or) process-flow view (see ERTmail thread): 1) Requirements capture => 2) Test specification => 3) Test => 4) Test result => 5) Collate results => 6) Analyze results (query) => 7) Present results
(view from 10,000 ft, to be expanded in following sections)
- largely a manual process in most contemporary TS efforts
- TA/TR identification, extraction, capture, etc
- TP synthesis (incl "generic" concept?)
- TC writing (TC description, TC content, ...)
- DOM TS as notable exception to "mostly manual" characterization
- IUT (content, API, implementation)
- inputs (validators, test-driver, content)
- test method: operator scripts, scripts, ...
- reference outputs (..., ..., images or text)
- result method: visual, automated comparison, ...
- result: verdict criteria, ...
- reporting
- methods
- collection/repository
- referencing from test cases (section; Xpointer; anchor)
- direct
- indirect, implied, generic
- handling discretion & optionality
- vague (cannot test)
- comprehensive
- defensible (traceable)
- to harness or not to harness (TS topology, navigation, etc)?
- progressive
- overall structure, design, phasing (BE, DT, ...)
- strong naming convention
- ...etc...
- atomic
- but not too atomic (consolidation of multiple atomic TCs, and why it is sometimes useful)
- some key-combinations
- xml-based TC description
- self-documenting
- accessible
- [...much more needed here...]
- per-TC instance
- and per TS release
- basic, manual (e.g., SVG);
- automated script-driven (e.g., XSLT)
- EARL (is this reports, or results? )
-...any?...
(Should this Framework document have a conformance clause? What would it say?)
The following QA Working Group and Interest Group participants have contributed directly to the content of this document: