Introduction· QA Framework audience· Primer·
QA Homepage· Latest News· QA Resources· QA WG· QA Handbook· Spec Guidelines·
QA Framework Primer & User Scenarios provides
a general orientation to the whole of the the QA
Framework. The QA Framework is a set of related documents (a
Recommendation, WG Notes, FAQ, Matrix, and Wiki) whose goal is to improve the
quality of specifications and the implementation of these specifications.
This Primer provides a roadmap to these documents. It introduces and orients
the reader by presenting the QA Framework from three different
prespectives - a document view, a role-based view, and a Working Group
milestones view.in the form of user and usage scenarios, primer
material, and target audience identification by Working Group role.
This document was previously published as a part of The QA Handbook — see section 6 of first published working draft of the said document.
QA Framework Primer is closely linked with and will continue to be coordinated with the other parts of the QA Framework: The QA Handbook; QA Framework: Specification Guidelines; Matrix, Test FAQ.; and some more informal materials about testing in the QA Activity Wiki.
Comments on this document may be emailed to www-qa-wg@w3.org, the publicly archived list of the QA Working Group. Commenters please note that your comments will be publicly archived and available, do not send information that should not be distributed, such as private data.
QA Framework Primer provides a general orientation to
the whole the QA Framework family of documents. The QA Framework
consists of a set of related documents (a Recommendation, WG Notes, FAQs,
Matrix, and Wiki) whose goal is to improve the quality of specifications and
the implementation of these specifications. This Primer provides a roadmap of
these documents. It introduces and orients the reader by presenting the QA
Framework from three different prespectives - a document view, a role-based
view, and a Working Group milestones view.
By presenting three views into the QA Framework, the reader is able to read and focus on areas of interest. The reader is encouraged to jump into any section of the document, reading only those sections relevant to the reader or reading the Primer in its entirety.
The target audience for this Primer is anyone and everyone with an interest in finding out about the QA Framework documents - to get a general understanding of the documents, to idenitfy which documents are of interest, and/or to know when to use which document.
The QA Framework consists of these existing documents:
In addition to these documents, the QAWorking Group also developed several templates to help implement good practices described in these documents. These templates are available (via links) from the QA Framework document to which it applies. In particular, there is a Charter template available from the QA Handbook and How to write a Conformance Clause template available from the Specification Guidelines document.
The following provides an overview of each document in the The QA Framework.
This document is a non-normative handbook about the process and operational aspects of certain quality assurance practices of W3C's Working Groups, with particular focus on testability and test topics. It is a a practical guide about applying good practices and quality assurance techniques to WG activities, especially developing Recommendations and test materials. It flags avoidable pitfalls, and provides techniques, tools, and templates that will facilitate and accelerate the work of the groups. Guidance in QAH is supported by real-world stories and examples. QAH will help Working Groups avoid known pitfalls and benefit from experiences gathered from the W3C Working Groups.
Specification Guidleines (SpecGL)
This document is a guide for editors of W3C specifications.It helps W3C editors write better specifications, by making a specification easier to interpret without ambiguity and clearer as to what is required in order to conform. It focuses on how to define and specify conformance. It addresses the most basic and critical topics with respect to conformance, including how to convey what is required for an implementation in order to conform. It also addresses how a specification might allow variation among conforming implementations. The document presents guidelines or requirements, supplemented with good practices, examples and techniques. As a result of using SpecGL, Working Groups will be able to produce specifications that are precise, easier to interpret without ambiguity or confusion, and clearer as to what is required in order to conform.
This document details some of the most important conformance-related concepts evoked in the QA Specification Guidelines, by developing some of the analysis that need to be considered while designing a specification and providing advanced techniques, particularly for dealing with conformance variability and complexity. This document analyzes how design decisions of a specification's conformance model may affect its implementability and the interoperability of its implementations. To do so, it introduces the concept of variability - how much implementations conforming to a given specification may vary among themselves - and presents a set of well-known dimensions of variability. Its goal is to raise awareness of the potential cost that some benign-looking decisions may have on interoperability and to provide guidance on how to avoid these pitfalls by better understanding the mechanisms affected by variability.
Test FAQ provides introductory information about the purpose of testing, how to get started, and what the testing process involves. This FAQ primarily documents what is already considered good testing practice or the norm, but it also includes a number of advanced testing goals that have not yet been fully achieved by any Working Group. The Test FAQ document is addressed to those who develop tests or organize testing efforts. It should also be useful to those who develop specifications or who run tests. It stresses early planning for testing, defines what makes a good test and examines test reporting, publishing and packaging of tests.
The Matrix is a table of W3C Specifications that are at at least Last Call stage (i.e., at Last Call, Candidate Recommendation, Proposed Recommendation or Recommendation). For each specification entry, a symbol indicates if the specification has a conformance clause and if there are conformance tools or test suites associated with the specification.
The Wiki is an open forum that allows anyone to contribute, share and develop QA ideas, tools and methods at W3C. Anyone is encouraged to add individual opinions, comments, and solutions to problems directly on the pages. Topics focus on writing good specifications and building test suites
This section identifies the parts of the The QA Framework that might interest those who fill the various roles within a Working Group or that interact with a Working Group. While each part of framework is targeted itself at a specific principal audience, the various parts might have somewhat broader interest and applicability.
This section looks at the QA Framework from the point of view of when during the life of a Working Group would a QA Framework document or section of a document be useful. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality assurance activities.
Because QA is properly an integral part of the activities of each Working Group, each Working Group has to consider and commit to a set of QA deliverables appropriate to its work. A spectrum of possibilities are discussed and illustrated in the early planning and commitment module of The QA Handbook.
If a Working Group is being newly formed, and if the it is able to anticipate and agree at Charter time on deliverables like test suites, then it should consider documenting those QA deliverables in its Charter, just as it does all other deliverables. Again, see the early planning and commitment section of The QA Handbook. A Working Group being re-chartered is a similar case to a newly formed group, although the scope and direction of its work might be clearer.
For an already-chartered Working Group undertaking new test and other QA-related activities, if these deliverables are not documented in the Charter already, then there are a couple of options. The W3C Process Document describes how to amend a Charter to accommodate significant new deliverables, if it wishes to take this route.
Once the Working Group is off and running, and assuming that it has planned on some test- or other quality-related deliverables, the next step is to chose and document the processes and logistics that it will use for its QA activities. These include such typical details as:
The sections within the Day-to-Day operations module of The QA Handbook give good-practice advice about how to do this, plus examples and a handy template for writing a QA Process Document.
There is a tight bond between how a specification is written on the one hand, and on the other hand its testability and its suitability as a basis for interoperable implementations.
QA Framework: Specification Guidelines should be applied from very beginning. Among the key topics that it addresses are:
Consider the advice of QA Framework: Specification Guidelines even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" and W3C Manual of Style, spec editors should refer to the spec guidelines throughout their work, on topics like testable language, clarity, conciseness.
A new Edition of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered a good time to align a specification to QA Framework: Specification Guidelines — the changes associated with such alignment could significantly disrupt and restructure the specification.
A new Version of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.
This stage in the specifications life has two significant aspects:
When the specification is published in TR space, then W3C Members not participating to the Working Group and the general public begin to review and comment. It would be valuable that such reviewers consult and understand the material in QA Framework: Specification Guidelines — it gives and informed set of evaluation criteria about the conformance, testability, and interoperability aspects of the specification.
Working Group participants and especially Test Task Force participants should refer to the good-practice pieces about advancement criteria and synchronization (between specs and test materials) of The QA Handbook. Projects enter The Matrix at Last Call Working Draft (if not sooner). A de-facto process convention is emerging, that there should be significant conformance test materials before finishing Candidate Recommendation phase. This timing coordinates with the explicit process requirement of two interoperable implementations.
There are several scenarios for how the Working Group "builds" its conformance test materials:
All these topics are addressed in more details in the Test FAQ.
Before starting the development, the Test Task Force participants would benefit from a familiarity with the material in the Test FAQ. and optionally in the articles of the QA Wiki regarding tests suites. There is useful information for both high-level planning — e.g., does the Working Group want breadth-first Basic Effectivity or a fully detailed suite? — as well as specific details for building the individual test cases.
Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the Test FAQ. with examples of review guidelines..
Several high-quality test suites have been developed outside of the relevant W3C Working Group, and then transferred to the group, or have been the result of assembling pieces from various contributions. Groups which are considering such a transfer should refer to test materials acquisition module of The QA Handbook. Clearly, the quality of the candidate test materials should be carefully assessed, but also, the licensing issues (as discussed in the QA Handbook needs careful planning.
Typically, a Working Group Test Task Force will want to publish releases of test materials, particularly as the specification advances through its final maturity levels (e.g., Proposed Recommendation) towards Recommendation.
One hurdle on the way to publication is legal — deciding and agreeing on suitable publication licenses. Advice on navigating this potential quagmire is presented in the licensing module of The QA Handbook.
As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions.
Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Imported test materials" are equally applicable here.
When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned . In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and applicability of particular tests . Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if the Working Group ceases active development of test materials, it may want to continue to consider submissions, and review and integrate them .
It is possible that the Working Group and its Test task force may disband after its charter expires. The various aspects of this situation are introduced in The QA Handbook .
The following QA Working Group and Interest Group participants have
contributed significantly to this document:
Last modified $Date: Wednesday 24 August 2005 - 08:00:52$ by $Author dom $