[techs] My notes for Quality Assurance Introduction, Operational Guideline and Test Guideline documents <very long>

To assist the techs group in the creation our documents I am reviewing
the W3C QA (Quality Assurance) specs. As a follow up we could review my
notes during one of our Wednesday calls.

 

Here are my notes for three of the QA documents. Since there are several
documents that I'll be reading, I'll just send notes as I finish a
couple. 

 

I'm reviewing seven documents plus the QA Process Document template. A
complete list of the documents is at the end of this mail. We may also
want to review the glossary [8] so we can begin to use their
terminology. Also at the end of this mail I'll reiterate all of the
questions so we can send them to the QA WG if necessary. I have a few
questions for our group as well, which I will list after the QA WG
questions.

 

The Introduction document outlines the seven documents that makeup the
QA framework. 

The title of document is "QA Framework: Introduction" [1]. 

The printed version is 16 pages.

The status is Working Group Note as of 12 September 2003.

 

The QA framework addresses three areas: specification editing,
production of test material, and coordination of QA efforts with
internal and external groups.

 

The three guidelines are Operational, Specification and Testing. Each of
these three documents has an accompanying document with examples and
techniques.

 

The introduction describes the target audiences and which documents they
should read. For example spec editors & authors should read the
Operational and Specification guidelines. Our lucky WG chairs would
benefit by reading all three. WG-TS participants will want to read the
specification and testing guidelines. (QA Question #1: References to TS
in this document are undefined and I assume are an acronym for Test
Suite?)

 

2.3 addresses structure and content, in which they have adopted WCAG
1.0's "guideline" and "checkpoint" terminology. With the additional use
of RFC 2119. RFC 2119 requires that certain keywords like MUST and MUST
NOT be used. For normative purposes these words are capitalized; any
instances of the keywords for non-normative purposes may remain in lower
case. For the complete list of keywords please refer to RFC 2119 [10].

 

The first step is to define the QA deliverables. Then appoint a QA
moderator, create a "/Test/" home page in the WG's web space, create an
-ts email discussion list, create WG process document for QA and discuss
IPR and test materials. (QA Question #2: Do we need to move our tech
mails to another e-mail list?)

 

The documents continues with information on: Planning and writing the
specification, reviewing and progressing the specification, designing
and building test materials, Publication of Test materials.

 

 

The Operational document outlines the "common operational framework for
building conformance test materials".

The title of document is "QA Framework: Operational Guidelines" [2]. 

The printed version is 43 pages.

The status is Candidate Recommendation as of 22 September 2003.

 

Section 1.5 presents a visual layout explaining the relationships
between the documents. 

 

Section 1.6 explains the layout of a guideline. It has a checkpoint
number, the checkpoint title, the conformance requirements, the priority
level, discussion, and E & T (Example & Techniques) for the checkpoint.
Some checkpoints have a related checkpoint section.

 

Section 1.7 describes checklists, as we used them in 1.0. 

 

This document begins to use the acronym TM which stands for Test
Materials.

 

Section 1.9 gives a chronological view of the guidelines [9]. This is
really cool! This is worth taking the time to read.

 

There are 8 Guidelines:

Guideline 1. Commit to Quality Assurance in Working Group activities. 

Guideline 2. Commit to resource level for Working Group QA activities. 

Guideline 3. Synchronize QA activities with the specification
milestones. 

Guideline 4. Define the QA process. 

Guideline 5. Plan test materials development. 

Guideline 6. Plan test materials publication. 

Guideline 7. Plan the transfer of test materials to W3C if needed. 

Guideline 8. Plan for test materials maintenance.

 

Guideline 1: Commit to Quality Assurance in Working Group activities.
WCAG WG Question #1: Does our charter define our commitment levels for
all three areas? It appears that we need to define A, AA or AAA for
Operational, Specification and Test. Guideline 1 also includes commit to
test materials, this doesn't have to be specific but what our intentions
are. Checkpoint 1.3 Priority 3 is interesting it says we must commit to
complete test materials specifically one test cases for every
identifiable conformance requirement. WCAG WG Question #2: Would the
total of test cases for us be: Total = checkpoints * technologies?
Checkpoint 1.4 Priority 1 requests that we enumerate QA deliverables and
expected milestones.

 

Guideline 2. Commit to resource level for Working Group QA activities.
This guideline has 3 checkpoints that are all priority 1. Address where
and how conformance test materials will be produced. Address QA staffing
commitments; at least appoint a QA point of contact. And request
resource allocation. 

 

Guideline 4. Define the QA process. 

This guideline we have pretty well covered. We have QA moderators
(Michael and Wendy), we have a Task Force (folks who call in Wed am).
WCAG WG Question #3: Do we need to complete the QAPD (QA Process
Document template) [7], create a mailing list just for test and create
the test web page? 

 

 

Guideline 5. Plan test materials development.

Define a framework for TM development

Ensure TM are documented for their intended purposes

Define a contribution process

Address license terms for submitted test materials

Define review procedures for submitted test materials

WCAG WG Question #4: Since I have just recently re-joined the group I'm
not sure where we stand on this.

 

Guideline 6. Plan test materials publication is about:

*         their support and maintenance includes a secure repository; 

*         tests materials are freely available for download; 

*         any licenses do not preclude the test suite from being used by
all interested parties; 

*         testing and publication of test results is encouraged.

 

Even though the intro says that a tech group like our does not need to
read this document, I have found it helpful and others might too. At the
minimum I'd suggest reading Guidelines 5-8.

 

The test document "defines a set of common guidelines for conformance
test materials".

The title of document is "QA Framework: Test Guidelines" [6].

The printed version is 18 pages.

The status is Working Draft as of 16 May 2003.

 

Before I share my notes, I'll list out some of the terminology used. The
complete list can be found in the glossary [8].

 

Atomic Test 

A test case that tests a single rule from the specification and maps
back to exactly one assertion. This is in contrast to some test cases
that may test a combination of rules.

 

Test Assertion 

A statement of behavior, action, or condition that can be measured or
tested. It is derived from the specification's requirements and provides
a normative foundation from which test cases can be built.

 

Test Case 

An individual test that corresponds to a test purpose, which in turn
maps back to the assertion(s), and finally the spec.

 

Test Suite 

A set of documents and tools providing tool developers with an objective
methodology to verify the level of conformance of an implementation for
a given standard

 

 

The details of the document are not complete. However, the guidelines
and checkpoints are complete. The QA WG wants comments on the guidelines
and checkpoints. There is not a document for the Test Examples and
Techniques yet.

 

The document addresses conformance test materials, test suites,
validation tools, conformance checklists, and other conformance
materials.

 

It is organized by guideline, checklist, and priority and uses RFC 2119.

 

There are six guidelines and 18 checkpoints:

1. Perform a functional analysis of the specification and determine the
testing strategy to be used.

2. Identify and tag testable assertions within the specification.

3. Provide a test management system

4. Provide a test framework

5. Provide results reporting

6. Plan for conformance testing

 

Checkpoint 1.1 Define the test suite scope [Priority 1]

Currently we have the techs documents for different technologies.
Additionally our scope could include test suites, test cases and test
pages. WCAG WG Question #5: Do we need to define our test suite scope?

 

Checkpoint 1.2 Identify the specification(s) to be tested. [Priority 1]

I'm interpreting this as testing and not editing. Editing would include
making sure that each checkpoint had a corresponding html test case -
does that sound right? WCAG WG Question #6: I guess this means we need
to say that the V2 will be tested.

 

Checkpoint 1.3 Analyze the structure of the specification, partition it
as appropriate, and determine and document the testing approach to be
used for the test suite as a whole and for each partition. [Priority 1]

 

Checkpoint 2.1 Identify and list assertions within the specification.
[Priority 1]

 

Checkpoint 2.2 Metadata must be associated with test assertions,
enabling test developers, the test-management system, the test-execution
framework, and the results-reporting process to make useful distinctions
between groups of tests. [Priority 2]

 

Checkpoint 3.1 The Working Group must provide a test management system
[Priority 1]

 

Checkpoint 3.2 The test management system must associate tests with
metadata [Priority 1]

 

Checkpoint 3.3 Test management system must allow filtering based on
metadata. [Priority 2]

 

Checkpoint 3.4 Test management system must support results. [Priority 2]

 

Checkpoint 4.1 Provide a test framework [Priority 1]

WCAG WG Question #7: Shall we discuss on Wed call and define our test
framework?

 

Checkpoint 4.2 Prototype the test framework [Priority 2]

This is a good idea, it also talks about platforms that it will support.
WCAG WG Question #8: We should also define our test matrix (OS x browser
x ATV device x ?) Keeping in mind that it is impossible to test 100%!

 

Checkpoint 4.3 Automation of testing is encourage [Priority 2]

 

Checkpoint 5.1 Test Materials must support results reporting [Priority
1]

 

Checkpoint 5.2 Results reporting framework should generate a unified
report [Priority 2]

 

Checkpoint 5.3 Results reporting framework must indicate result status
of each test [Priority 1]

Reporting what needed to be reviewed by a person would be fabulous!

 

Checkpoint 5.4 Results reporting framework should allow filtering based
on metadata [Priority 2]

 

Checkpoint 6.1 Organize conformance testing activities [Priority 1]

 

Checkpoint 6.2 Encourage Vendors to publish test results [Priority 3]

 

This guideline requires an assertion about conformance.

 

A possible next step for us could be to take one checkpoint from the 2.0
draft and work it through the test guideline. We could start with
"Guideline 1.1 For non-text content, provide text equivalents that serve
the same purpose or convey the same information as the non-text content,
except when the purpose of the non-text content is to create a specific
sensory experience (for example, music and visual art) in which case a
text label or description is sufficient." Principle should be referred
to as "Guideline" and Guideline should be referred to as "Checkpoint".
And it does not use a RFC 2119 keyword. Next step would be to identify
the technologies this applies to. And write the technique part of it,
then write the test cases. This may help us to determine what is missing
from our framework. I have a few folks who are new to accessibility
testing that could try out the framework. 

 

Questions for QA WG

QA Question 1: References to TS in this document are undefined and I
assume are an acronym for Test Suite?

QA Question #2: Do we need to move our tech mails to another e-mail
list?

 

Questions for our WG

WCAG WG Question #1: Does our charter define our commitment levels for
all three areas?

WCAG WG Question #2: Would the total of test cases for us be, Total =
checkpoints * technologies?

WCAG WG Question #3: Do we need to complete the QADP(QA Process Document
template)[7], create a mailing list just for test and create the test
web page?

WCAG WG Question #4: Since I have just recently re-joined the group I'm
not sure where we stand on this.

WCAG WG Question #5: Do we need to define our test suite scope?

WCAG WG Question #6: I guess this means we need to say that the V2 will
be tested.

WCAG WG Question #7: Shall we discuss on Wed call and define our test
framework?

WCAG WG Question #8: We should also define our test matrix (OS x browser
x ATV device x ?)

 

I hope the tech group finds my notes helpful, please let know if you
have any question.

Thanks,

Jenae

 

Seven QA Framework documents plus the glossary:

1. QA Framework: Introduction [1]

2. QA Framework: Operational Guidelines [2]

3. QA Framework: Operational Examples & Techniques [3]

4. QA Framework: Specification Guidelines [4]

5. QA Framework: Specification Examples and Techniques [5]

6. QA Framework: Test Guidelines [6]

7. QA Process Document template for "QA Framework: Operational
Guidelines" [7]

8. Glossary [8]

 

 

[1] http://www.w3.org/TR/qaframe-intro/ 

[2] http://www.w3.org/TR/qaframe-ops/ 

[3] http://www.w3.org/QA/WG/2003/09/qaframe-ops-extech-20030912 

[4] http://www.w3.org/TR/qaframe-spec/ 

[5] http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech-20030912 

[6] http://www.w3.org/TR/qaframe-test/ 

[7] http://www.w3.org/QA/WG/2003/09/OpsET-qapd-20030912 

[8] http://www.w3.org/QA/glossary

[9] http://www.w3.org/TR/qaframe-ops/introduction#chronological-view 

[10] http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2119.html