W3C

W3C Quality Assurance Framework

W3C Working Draft 06 Nov 2001

This version:
http://www.w3.org/QA/2001/Note-qa-framework-20011106
Latest version:
http://www.w3.org/QA/qa-framework
Previous version:
This is the first draft.
Editor:
Lofton Henderson (lofton@rockynet.com)

Abstract

This document defines a common framework for building conformance test suites and tools for W3C specifications. It presents organizational, operational, and procedural guidelines for groups undertaking conformance materials development, as well as technical guidelines for the test suites (TS) and test tools themselves.

Status of this document

This document is a Note, made available by the W3C Quality Assurance Activity (QA) for discussion at the first QA face-to-face meeting.

This version is a first, incomplete draft. Some sections are written, others are only outlined. Some outlines could be considered fairly complete, but others -- particularly those in section 4 -- are not. Open issues and other incompletions are flagged with "@@". In different sections, for illustrative purposes, the style varies from prose, to Guidelines, to Guidelines-plus-Checkpoints, to the latter plus proposed external references to "Examples & Techniques". In this document version, guidelines and checkpoints are not yet numbered or formatted, and the proposed "Techniques" external references are in fact illustrated by internal blocks of text (not external content).

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org.

Table of contents

1. Document goals and scope
1.1 Goals
1.2 Scope
1.3 Applicable domain
1.4 Document orientation & structure
1.5 Level & target audience
1.6 Terminology
2. Organizational considerations
2.1 W3C process requirements
2.2 De facto QA process conventions
2.3 Where to do TS development
2.3.1 Test materials home
2.3.2 Where to build the suite
3. Procedural & operational framework
3.1 Principal aims of TS development
3.2 WG Charter considerations
3.3 In-WG process
3.3.1 Starting & synchronizing with spec progression
3.3.2 First, Create a QA (Test) Process Document
3.3.3 Define and setup logistics
3.3.4 Who (staffing roles)
3.3.5 Build requirements
3.3.6 Contribution requirements
3.3.7 Repository, version, and access control
3.3.8 IPR and copyright
3.3.9 Distribution & access considerations
3.3.10 Accessibility & Internationalization requirements/harmonization
3.3.11 Maintenance procedures
3.4 WG relationship to QA Activity
3.5 Transferring a TS from outside W3C
3.6 Logos & branding
3.7 other Ops&Proc topics
4. Technical Framework
4.1 Caveat about taxonomy dependencies
4.2 Idealized testing methodology (in 4-5 steps)
4.3 Idealized TS-build methodology (in 4-5 steps)
4.4 TS components (taxonomy-sensitive)
4.5 TR/TA identification or extraction
4.6 TP synthesis or derivation
4.7 Principles of Test Suite design & content
4.8 Principles of Test Case design & content
4.9 Serialization/versioning
4.10 Results and reporting
4.11 User documentation
4.12 Other technical topics
5. Conformance
6. Acknowledgments
7. References


1. Document goals and scope

1.1 Goals

This document defines a common framework for building conformance test materials for W3C Recommendations. It presents organizational, operational, and procedural guidelines for groups undertaking conformance materials development, as well as technical guidelines for the test suites (TS) and tools themselves.

Foremost amongst the purposes of defining a common framework is the principal goal of QA development itself:

More specific goals of this framework document include:

Ideally, this framework document would provide to those undertaking test suite projects,

although the degree to which a simple cookbook guide is achievable is complicated by the diversity of potential test suites and tools, as well as by organizational diversity.

As a W3C Note, this document aims to serve as a vehicle and framework for discussion within the W3C Quality Assurance (QA) activity, in the fora of the QA Working Group [QAWG] and QA Interest Group [QAIG].

This does not preclude that portions of these guidelines may end up with a status other than Note, either by incorporation into the W3C Process Document [PROCESS], or by processing in the W3C Recommendation track.

1.2 Scope

The scope of this document encompasses:

Outside the scope of this document are:

The QA Activity considers W3C specification guidelines -- on topics including clear conformance clauses, enumeration of discretionary behaviors, and identification and addressability of testable assertions -- to be a critically important part of the QA programme of work. They will be addressed in a separate document [@@ref, when there is one].

1.3 Applicable domain

Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:

This framework and its guidelines should be applicable and useful throughout this diverse context.

1.4 Document orientation & structure

This is not intended to be a survey of existing W3C practice. The QA matrix [MATRIX] is a survey of all known existing and in-progress conformance test materials for W3C standards. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.

As much as possible, the framework will be presented as a set of informative guidelines for WG members and TS developers. Some of the WAI standards, such as WCAG [WCAG10], provide a good model of such a guideline-oriented style, using verifiable checkpoints

To deal with the complexities noted in the preceding section, the guidelines of this document will as much as possible be stated with general, cross-taxonomy applicability. A companion document, "QA Framework Examples and Techniques" [@@EX-TECH], will have additional detailed content linked from some of these individual guidelines. The linked material will explore, define, give examples of taxonomy-dependent details, as well as present techniques for implementing guidelines.

Although these are referred to as "informative guidelines", it is recognized that some of this content may one day become normative -- one of the meta-standards for development of W3C standards.

1.5 Level & target audience

This framework specification aims to be comprehensive -- it spans the breadth of organizational, process, and technical topics which test suite developers are likely to encounter. While it should be useful across a range of reader experience levels, it is an explicit goal to be useful and accessible to readers at the start of QA requirements planning or development of test suite materials.

Languages for test case definition (TCDL), test results expression (e.g., EARL), are an important part of a conformance test framework. In this specification, they will be briefly mentioned in context. However their examination in any detail will be reserved for later specifications -- Advanced Topics.

The principal intended audience is the members of new and existing Working Groups (WGs), especially those who have interest in or will be involved in the QA activities of their WG. These will include test-suite builders, managers/editors, and test case contributors.

1.6 Terminology

One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in this framework specification are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.

(@@Ed. note -- following is not uniformly implemented yet.)

Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.

2. Organizational considerations

2.1 W3C process requirements

This section is informative background -- there are no guidelines in here.

The current W3C Process Document says virtually nothing about conformance testing. Two mentions may be found:

  1. "[...]groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (sec 3.)
  2. "W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing testbed applications, helping to create test suites, etc.)" (sec 4.)

No other process documents yet address the requirements or processes of building test suites and tools. One identified deliverable of this QA activity may be changes to the W3C Process Document, or production of other normative, QA-specific process documents.

2.2 De facto QA process conventions

Nevertheless, there is a body of contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and validators, in various states of development.

A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

Therefore, based on contemporary practice and coordination with related activities, a sensible process requirement is:

Guideline. Working Groups should consider completion and publication of significant test materials to be a criterion for CR-exit and PR-entrance.

2.3 Where to do TS development

2.3.1 Test materials home

Guideline. Test suites and tools should ultimately reside in W3C.

Checkpoint. W3C should provide secure and reliable repository location for the test materials.

Checkpoint. W3C should have ultimate responsibility -- though not necessarily sole responsibility -- for the ongoing evolution of test suites and tools, after some milestone of operational maturity of the materials.

Explanation & Rationale. This does not necessarily mean that the WG must solely build the test suites and tools -- in fact there are good reasons why external parties should have a strong participation during the building of the materials (see @@below). But once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), they should reside in and be the responsibility of W3C, and in particular the Working Group responsible for the subject standard. Reasons:

  1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications (RECs) normally increase, and the WG is the best venue for initial processing and adjudication of such queries against both.
  2. Past experience suggests that the WG is more likely to have better longevity than external, ad-hoc (dedicated to the specific test materials tasks) entities.
  3. Further to #1, it is more likely that specification (REC) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the WG.

If the WG responsible for a standard becomes defunct (@@does this ever happen, for a REC that still stands?), then another home for the test materials would be found within W3C. Those responsible for ongoing interpretation and maintenance (errata) of the standard should be designated.

2.3.2 Where to build the suite

Existing test materials developments show several patterns:

  1. W3C model. The whole process, from start to deployment, is done in the WG or an affiliated -ts group Example: SVG test suite [SVGTEST].
  2. External model. The test suite is built somewhere else, and transferred to W3C. Example: XML test suite [XMLTEST] was built in the OASIS XML Conformance TC and is being transferred to W3C.
  3. Mixed model a. Initial "seed" work is done elsewhere, and transferred to W3C. Example: none yet. (@@anyone know of any such examples?)
  4. Mixed model b. Test suite is built by a collaboration between external group and W3C, where substantial parts of the work are performed by each. Example: none yet. (@@anyone know of any such examples?)

The determining factors for a new test suite effort will be:

Building test suites is expensive. See below. (@@link to end of "WG Charter...", its EX-TECH reference, or even directly into that EX-TECH reference from here.)

The decision might not always be W3C's alone. In several past cases, an outside group has taken the initiative and begun work. Example: XSLT test suite and tools [XSLT-TEST] are being built by the OASIS XSLT/Xpath Conformance TC.

Guideline. The TS development process benefits from the involvement of both WG-member experts on the subject standard, and neutral participants who are uninvolved in writing the standard.

Explanation & Rationale. Regardless of the development pattern -- which organizations do the development work -- experience has shown that there is high value in having active participation by parties other than the people who wrote or are writing the REC. The general benefits of test suite work are:

These benefits pertain, to some degree, regardless of where the work is done. However, in a QA mail thread [EXTERN-TA] it is asserted, and there is apparent consensus based on test suite experiences, that the participation of third parties helps to detect some kinds of spec-testability problems that may be invisible to the writers of the subject standard.

3. Procedural & operational framework

3.1 Principal aims of TS development

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

More specific goals of test materials development should include:

The goals of a common W3C QA framework do not encompass:

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, test suites and materials development should not arbitrarily preclude:

Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any braning-related goals are articulated.

3.2 WG Charter considerations

(@@Ed. note: We're trying this section in the style of WAI Guidelines/Checkpoints.)

Guideline. Working Group charters should address goals and plans for test suites and tools.

Explanation & Rationale. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.

Checkpoint. New WG charters SHALL address test suite plans.

Checkpoint. Existing WG charters SHOULD (MAY?) be amended with test suite plans.

Checkpoint. Charters should express commitment to general intent and plans.

Explanation. Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK]. A range of possible commitments has been identified. See "Examples & Techniques" [@@EX-TECH].

[...in EX-TECH, we would have:

The range of possibilities for WG commitment is:

0.) WG makes no firm commitment to address quality (not an option). 1.) WG promises to think about testability when writing their documents. 2.) WG makes an effort to have QA expertise among their membership. 3.) Beyond a few normative examples, WG aims to have numerous normative cases in the body of the Rec. 4.) WG commits that a test suite, not necessarily complete and thorough, will exist somewhere before they go to Rec. 5.) WG commits that they review and fully approve a test suite, not necessarily complete and thorough, before they go to Rec. 6.) WG commits to reviewing test cases on a continuing basis, including after going to Rec. 7.) WG will attempt to generate a complete catalog of test cases before going to Rec. 8.) WG insists on a complete catalog and partial implementation of a test suite before going to Rec. 9.) WG insists on a test suite, not necessarily complete and thorough, and provides an official test harness, before going to Rec. 10.) WG commits to writing test cases themselves and delivers a test harness and suite at some point. 11.) WG insists on a complete test suite before going to Rec.

This 12-point enumeration is derived from proposal to the QA mail list, after the 4/2001 QA Workshop. Point 0 is not an option -- WGs are required to commit to address quality at some level of effort.

The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]

Checkpoint. The WG charter should address where and how conformance test materials will be produced.

Explanation. There are several possibilities for producing conformance test materials:

Checkpoint. The WG charter TS plans should address staffing commitments.

Explanation. There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].

[...in EX-TECH, we would have:

-- quick tabular summary of size-of-effort data, from section 2.3 of http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.

-- quantify some examples, e.g., estimated XML uptake effort?

/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]

3.3 In-WG process

3.3.1 Starting & synchronizing with spec progression

- (charter redux: outlines and commits to the WG plan and intentions at general level)

- start: the sooner, the better

- early benefits of, e.g., TA analysis

- when to start TS work (WD, CR, ...)

- coordination with spec (REC) progression

3.3.2 First, Create a QA (Test) Process Document

- WG produce a QA (Test) Process Document addressing...

(bullet list here, each to be expanded in following sections)

- examples: DOM; SVG (virtually a process doc: testManual + cvsManual),

- (distill set of guidelines and checkpoints from the SVG & DOM examples).

3.3.3 Define and setup logistics

- mail lists (make a -ts list),

- web page intro (../Test); examples: SVG, DOM.

- ...etc...

3.3.4 Who (staffing roles)

- manager/moderator/editor (3 terms for same thing)

- some WG staff (and other W3C members)

- some QA staff?

- some 3rd party (e.g., for TR analysis)?

3.3.5 Build requirements

- content goals (BE, DT, numbers, etc)

- decide and document (technical) process & methodology

- define review/acceptance procedures and criteria

3.3.6 Contribution requirements

- define Submission Requirements

- define review & acceptance criteria (probably same as above)

- plan (identify) tools, methods, & infrastructure for accepting and integrating

3.3.7 Repository, version, and access control

- extract guidelines from SVG (look also at DOM TS Process).

3.3.8 IPR and copyright

- on contributed materials

- on the TS itself

3.3.9 Distribution & access considerations

- packaging

- procedures

- public & user documentation

3.3.10 Accessibility & Internationalization requirements/harmonization

- WCAG requirements for content ("A" minimal guideline)

- ATAG or UAAG requirements for tools

- I18n considerations?

- pubrules and documents

3.3.11 Maintenance procedures

- spec: errata (Process Doc? Practical?)

- TS correctness challenges

- resolving

- fixing

- appeals (principles/guidelines from email threads)

3.4 WG relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.

Potential relationships between Working Groups and the QA Activity include:

Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

3.5 Transferring a TS from outside W3C

- parties must to sort out IPR

- charter amendment? yes.

- need to sort out new home (WG), staffing, and other issues at "Charter" level of detail.

(several guidelines and checkpoints can be distilled from XML transfer example)

- after the transfer, all of the above, "In-WG", stuff pertains.

3.6 Logos & branding

- WAI does this for content; CSS also and basic HTML validation.

- Controversial topic, as email thread and UAAG CR comments show.

- Taxonomy dependent: content (WCAG, CSS, etc) is less difficult/controversial; UAs will be contentious.

3.7 other Ops&Proc topics

- ...any?...

4. Technical Framework

(Right, this is a real hodge-podge still. Very little development yet, as I've focused on previous sections. Main issues will be similar -- prosey vs. guidelines? separate document for details, like WAI "Techniques"? Taxonomy dependencies will be more acute here, as illustrated by: HTML validator; DOM TS; SVG TS)

4.1 Caveat about taxonomy dependencies

- for content&methodology

- will impose per-family splits in some of the following technical guidelines

- try to articulate a guideline at a general level

- handle details with companion document, "Examples & STechniques".

4.2 Idealized testing methodology (in 4-5 steps)

- several potential views

- tester activity view: configure ... test ... judge ... record ... report

- (or) data view: TCDL ... TC ... { Infosetized XML | EARL }

- (or) ??? view: (what view is most useful?).

- (or) process-flow view (see ERTmail thread): 1) Requirements capture => 2) Test specification => 3) Test => 4) Test result => 5) Collate results => 6) Analyze results (query) => 7) Present results

4.3 Idealized TS-build methodology (in 4-5 steps)

(view from 10,000 ft, to be expanded in following sections)

- largely a manual process in most contemporary TS efforts

- TA/TR identification, extraction, capture, etc

- TP synthesis (incl "generic" concept?)

- TC writing (TC description, TC content, ...)

- DOM TS as notable exception to "mostly manual" characterization

4.4 TS components (taxonomy-sensitive)

- IUT (content, API, implementation)

- inputs (validators, test-driver, content)

- test method: operator scripts, scripts, ...

- reference outputs (..., ..., images or text)

- result method: visual, automated comparison, ...

- result: verdict criteria, ...

- reporting

4.5 TR/TA identification or extraction

- methods

- collection/repository

- referencing from test cases (section; Xpointer; anchor)

4.6 TP synthesis or derivation

- direct

- indirect, implied, generic

- handling discretion & optionality

- vague (cannot test)

4.7 Principles of Test Suite design & content

- comprehensive

- defensible (traceable)

- to harness or not to harness (TS topology, navigation, etc)?

- progressive

- overall structure, design, phasing (BE, DT, ...)

- strong naming convention

- ...etc...

4.8 Principles of Test Case design & content

- atomic

- but not too atomic (consolidation of multiple atomic TCs, and why it is sometimes useful)

- some key-combinations

- xml-based TC description

- self-documenting

- accessible

- [...much more needed here...]

4.9 Serialization/versioning

- per-TC instance

- and per TS release

4.10 Results and reporting

- basic, manual (e.g., SVG);

- automated script-driven (e.g., XSLT)

- EARL (is this reports, or results? )

4.11 User documentation

4.12 Other technical topics

-...any?...

5. Conformance

(Should this Framework document have a conformance clause? What would it say?)


6. Acknowledgments

The following QA Working Group and Interest Group participants have contributed directly to the content of this document:

7. References

EXTERN-TA
QA activity email thread about third-party participation in test materials production.
MATRIX
W3C-wide conformance activity survey covering all the WGs, "The Matrix".
PROCESS
W3C Process Document, 19 July 2001.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials.
QAIG
Quality Assurance Interest Group of the W3C QA Activity.
QAWG
Quality Assurance Working Group of the W3C QA Activity.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997.
SVGTEST
SVG Working Group's test suite resource page.
WCAG10
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
WG-QA-RANGE
Email proposal by David Marston, on the QA public mail list, for range of WG commitment levels to conformance test materials production.
XMLTEST
OASIS XML Conformance TC's XML test suite resource page.
XSLT-TEST
OASIS XML Conformance TC's XSLT/Xpath test suite resource page.
...ETC...
...etc...etc...etc...