http://www.w3.org/TR/qaframe-test/ QA Framework: Test Guidelines http://www.w3.org/TR/2002/WD-qaframe-test-20021220/ 5 Kirill Gavrylyuk (kirillg@microsoft.com) Dimitris Dimitriadis (dimitris@ontologicon.com) Lofton Henderson (lofton@rockynet.com) Mark Skall (mark.skall@nist.gov) Peter Fawcett (pfawcett@real.com) Contradictory behaviors Two or more different behaviors unconditionally prescribed by a specification for the same class of implementations under the same circumstances. Discretionary choices A value or behavior may be chosen from a well-defined enumerated set of two or more possibilities. Optional behaviors A well-defined feature may be supported or not (if supported, then the requirements are clear and unambiguous). Explicitly undefined behaviors Specification states that it is open ended and undefined, what set of values an element or attribute may have, or the behaviors of a product that implements a feature. Test Assertion A set of premises that are known to be true by definition in the spec. Test Area A minimal compound unit in the test suite structure. Test Framework A set of utilities, stylesheets and documentation that describe and facilitate development, documentation and use of the tests. Results Verification A common testing practice used to determine if a test passes or fails by verification of the test result or output against the expected one. http://www.w3.org/TR/qaframe-intro/ QA Framework: Introduction http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/ 4 Lofton Henderson (lofton@rockynet.com) Kirill Gavrylyuk (kirillg@microsoft.com) Dimitris Dimitriadis (dimitris@ontologicon.com) Lynne Rosenthal (lsr@nist.gov) all WG members For any (potential) WG member, the charter and QA-commitment parts of Operational Guidelines ([QAF-OPS], Guideline 1) should be helpful in understanding what the WG has committed to deliver. Familiarity with the Specification Guidelines[QAF-SPEC] will be helpful to any member who participates in the advancement of the WG's specifications to Recommendation. WG spec editors & authors As for all WG members, the operational guidelines [QAF-OPS] are useful. A good working understanding of specification guidelines [QAF-SPEC] will be needed in order to satisfy the specification guidelines and checkpoints, and Specification Examples & Techniques[SPEC-EXTECH] should be a valuable resource in choosing document structure, formats, and techniques that will facilitate satisfying the requirements. WG chair As the person ultimately responsible for both the advancement of the WG's specifications and the WG's QA projects, a familiarity with the guidelines for operations and process [QAF-OPS], for specifications [QAF-SPEC], and for test materials [QAF-TEST] will be useful. WG-TS participant Those who are active in building the conformance test materials of the WG will need a good working understanding of both the guidelines for test materials [QAF-TEST] and Test Examples & Techniques[TEST-EXTECH]. Because of the close dependency of test materials on the functional specifications, a good familiarity with the specification guidelines [QAF-SPEC] as well as examples and techniques [SPEC-EXTECH] will also be helpful. WG-TS moderator The person who manages the WG's QA projects should have good working understandings of guidelines and techniques for specifications ([QAF-SPEC] and [SPEC-EXTECH]), as well as the guidelines and techniques for test materials ([QAF-TEST] and [TEST-EXTECH]). In addition, knowledge of all of the process and logistical setup guidelines [QAF-OPS] and techniques [OPS-EXTECH] are needed. non-WG spec reviewers Whether from other WGs, or the public at large, the specification guidelines [QAF-SPEC] will be helpful to those who review a WG's specifications. non-WGTM reviewers Whether from other WGs, or the public at large, reviewers of the test materials of a WG should be familiar with guidelines for test materials [QAF-TEST], and familiarity with the techniques [TEST-EXTECH] as well would facilitate a critical review. Reviewers of activity proposals & charters For those W3C members who will be reviewing Activity proposals and proposed WG charters, and helping to form their AC representatives positions, the commitment and deliverable requirements defined in the operational guidelines [QAF-OPS] are helpful. QA Activity members Members of the QAWG are an expert resource for the W3C Working Groups, and accordingly should be expert on all parts of the Framework; members of the QAIG need thorough familiarity with all parts as well, to effectively render some of the IG's chartered deliverables. http://www.w3.org/TR/qaframe-spec/ QA Framework: Specification Guidelines http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/ 5 Dominique Hazaël-Massieux (dom@w3.org) Lofton Henderson (lofton@rockynet.com) Lynne Rosenthal (lynne.rosenthal@nist.gov) Dimitris Dimitriadis (dimitris@ontologicon.com) Kirill Gavrylyuk (kirillg@microsoft.com) discretionary choices a value or behavior may be chosen from a well-defined enumerated set of two or more possibilities; optional features a well-defined feature may be supported or not (if supported, then the requirements are clear and unambiguous) implementation dependent values (or features) it is open ended and undefined, what set of values an element or attribute may have, or the behaviors of a product that implements a feature, etc conformance clause a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance conformance level a variety of conformance designation. Other designations include conformance category, conformance degree, conformance xxx,... "Conformance level" is discouraged in new specifications, because of confusion with "functional level". deprecated An existing feature that has become outdated by a newer construct or is no longer viable. Deprecated features should not be used and may be removed in some future version. dimensions of variability the ways in which different products that are conformant to a specification may vary among themselves. In this Specification Guidelines document, the dimensions of variability are used to help organize, classify and assess the conformance characteristics of W3C specifications discretionary items deliberate and explicit grants of discretion by the specification, to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc. functional level a technology subset that is one of a hierarchy of nested subsets, ranging from minimal or core functionality to full or complete functionally. implementation conformance statement (ICS) a mechanism for providing standardized information about an implementation of a named specification, usually in the form of a questionnaire on which product implementers report about the product's conformance to the specification. An ICS is used to indicate which requirements, capabilities and options have and have not been implemented. informative text text in a specification whose purpose is informational or assistive in the understanding or use of the specification, and which contains no test assertions or conformance requirements. level a commonly used shorthand for functional level. module a collection of semantically-related elements, attributes, and attribute values that represents a unit of functionality. Modules are non-hierarchical, discrete divisions that are defined in coherent sets. normative text text in a specification which is prescriptive or contains conformance requirements. profile a subset of a technology that is tailored to meet specific functional requirements of a particular application community. A profile may address a single technology; or, a profile can also group a set of technologies (i.e., from different specifications) and define how they operate together. Profiles may be based on hardware considerations associated with target product classes, or they may be driven by other functional requirements of their target communities. profiling a method for defining subsets of a technology by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. strict conformance conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented). test assertion a statement of behavior, action or condition that can be measured or tested(See also QA Glossary [QA-GLOSSARY].) unconditional conformance use case a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform. usage scenario an instance of a use case, that represents a single path through the use case. Thus, there may be a scenario for the main flow through the use case and another scenarios for each possible variation of flow through the use case (e.g., representing each option).