The XML Schema WG Task Force
on QA Framework documents: Review Report

Content

Task Force

QA Framework: Introduction
W3C Working Draft 10 February 2003
http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/

Note INTR-LA1.

Context

general

Comment

QA Framework documents use terms conformance, interoperability but do not define them. In many cases interoperable implementation is used instead of conformant implementation which is not obviously the same. For example, two implementations can be interoperable but non-conformant. And two conformant implementations can be non-interoperable.

Proposal

Define the terms "conformance", "interoperable implementations" or provide a reference to the document where they are defined.

Note INTR-LA2.

Context

2.2 Technical assets, bullet list

Comment

The section lists a collection of assets useful in QA. It misses though such important kind of tools as coverage measurement tools.

Proposal

Add to the list: "Methods and tools for test coverage estimation."

Note INTR-JT1.

Context

Section 2

Comment

The acronym "WAI" is used before being defined.

Proposal

Provide a definition of the "WAI" acronym.

Note INTR-JT2.

Context

Section 2

Comment

The last resource in the bulleted list simply reads "assets". I think the document would benefit from some clarification here.

Proposal

Clarification is needed.

Note INTR-LA3.

Context

3.4 Terminology "This document contains no normative requirements."

Comment

If the document doesn't contain any normative requirements, does it make sense to call it a specification then?

Proposal

Change "specification" to "document" where necessary (at least in Abstract and Status sections) .

Note INTR-JT3.

Context

Section 4.2.3

Comment

It is not clear whether the bulleted list supplied is a list of general topics or names of sections in the Specification Guidelines. If the latter, they should be capitalized and linked. If the former, more explanation of what is intended is required.

Proposal

Clarification is needed.

QA Framework: Operational Guidelines
W3C Working Draft 10 February 2003
http://www.w3.org/TR/2003/WD-qaframe-ops-20030210

Note OGL-LA1.

Context

1.1 Scope and goals. "In this guidelines document, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance."

Comment

It isn't clear what conformance this sentence is talking about.

Proposal

Explain or at least add: "...or indicate conformance of a implementation to a specification."

Note OGL-LA2.

Context

Checkpoint 1.1.

Comment

The QA level system is confusing. At least three different metrics are used: priorities of checkpoints (P1, P2, P3), degrees of conformance (A, AA, AAA) and QA levels (1..7). The level systems seems unnecessary and can be covered with priorities and degrees. Actually degrees directly correspond to priorities so theoretically one of them can be dropped too.

Proposal

Drop QA level system and use priorities instead.

Note OGL-LA3.

Context

Checkpoint 1.1.

Comment

the checkpoint is of P1 and requires to commit to QA level three which in its turn refers to checkpoints of P1, P2, P3 priority. If so, those tasks should P1 too.

Proposal

Revise priorities of checkpoints required by 1.1. Explicitely lists all other checkpoints required by different QA levels.

Note OGL-LA4.

Context

Guideline 3. Synchronize QA activities with the specification milestones. 3rd bullet: "it provides clear set of testable assertions - skeleton of the specification - which in its turn facilitates development of interoperable implementations. The latter is a W3C process criterion for entering the Candidate Recommendation phase. "

Comment

Good specification provides a base for development conformant implementation first of all. Interoperability means different. For example, two implementations can be interoperable but non-conformant. And two conformant implementations can be non-interoperable.

Proposal

Change "of interoperable implementations" to "of conformant and interoperable implementations"

Note OGL-LA5.

Context

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Comment

Unsynchronized levels of errata used in specs and test suites is an easy and most common way to non-conformant and non-interoperable implementations. This is absolutely necessary to indicate which version of spec and errata level the tests corresponds. Priority 3 is not sufficient to enforce this.

Proposal

Change priority to 1.

Note OGL-LA6.

Context

Checkpoint 4.3. Produce the QA Process Document. [Priority 1], 2nd bullet: "minimally addresses all of the topics required by other checkpoints in these specification guidelines. "

Comment

"minimally address" sounds ambiguous. It can be read as "at least" or "very little".

Proposal

Change to "as minimum address" or "at least minimally" or delete "minimally" at all.

Note OGL-LA7.

Context

Checkpoint 4.5. Define the QA framework for test materials development. [Priority 1] "Discussion. The QA framework describes how to develop, document and use the tests." also in Guideline 5. Plan test materials development. "a Working Group needs to have clear understanding of what QA framework it will use and how to insure the quality and usability of the test materials themselves."

Comment

The term "QA framework" is used in a different sense then in the titles of QA guidelines: "QA Framework: Operational Guidelines"

Proposal

Change "QA framework" to "QA TM framework" or "TM framework" ( "TS framework"? "TS design plan"?)

Note OGL-LA8.

Context

Checkpoint 4.5. Define the QA framework for test materials development. [Priority 1]

Comment

It's not clear why this is so important (P1). For example, a test suite can be constructed by combining several external test suites with different frameworks.

Proposal

Decrease priority to P2.

Note OGL-LA9.

Context

Checkpoint 5.4. Define review procedures for submitted test materials. [Priority 2] "...the procedure MUST minimally address criteria for accuracy, scope, and clarity of the tests."

Comment

"minimally" is ambiguous.

Proposal

Change to : "...the procedure MUST at least minimally address..." or "...the procedure MUST as minimum address..."

Note OGL-LA10.

Context

Checkpoint 6.2. Define the licenses applicable to published test materials. [Priority 1]

Comment

The discussion section contains some statements we have found difficult to understand clearly. In some cases, it recommends using the W3C Document License. However, the License was changed on 31 December 2002 to remove the grant of 'use'; some WG members believe that this renders the Document License inappropriate for Test Materials and have been instructed by their legal departments not to use test materials issued under the Document License as modified.

Proposal

The document could recommend one or more license types more appropriate for test cases, test suite documentation, executable test-suite software, and other test materials. We believe these licenses should provide explicit rights at least to use, copy, and distribute test materials, and limited rights to adapt test suites for example to accommodate requirements of a particular execution environment or a test automation system.

Note OGL-LA11.

Context

Checkpoint 6.3. Describe how and where the test materials will be published. [Priority 2] Bulleted list in Discussion section.

Comment

Arguments against publishing TM in the TR space and not convincing. The first bullet says that documents in TR space cannot be changed. This equally applicable to both specs and testsuites. Both are changed to accommodate errata. Besides archived versions of TM archives can be kept unchanged. The first bullet is true for source TM repositories not for archived and frozen versions.

Proposal

Change "Test materials change and evolve frequently even after the specification becomes W3C Recommendation." to "Repositories with test material are updated and evolve frequently even after the specification becomes W3C Recommendation. ="

Note OGL-LA12.

Context

Checkpoint 6.5. Address the publication of test results for products. [Priority 2] "Such publication should include or describe a test harness that would allow anyone to reproduce the results. "

Comment

In some cases test result may be produced using some non-public test harnesses. The harness also can be complicated enough to be easily used by anyone. It could be more important to describe exact environments and product's mode which those results have been produced for.

Proposal

Make it less strong. For example, remove the word "anyone": "Such publication should include or describe a test harness that would allow to reproduce the results. "

Note OGL-LA13.

Context

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 2]

Comment

Unsynchronized levels of errata used in specs and test suites is an easy and most common way to non-conformant and non-interoperable implementations. This is absolutely necessary to indicate which version of spec and errata level the tests corresponds. Priority 2 is not sufficient to enforce this.

Proposal

Change priority to 1.

QA Framework: Test Guidelines
W3C Working Draft 20 December 2002
http://www.w3.org/TR/2002/WD-qaframe-test-20021220

Note TGL-LA1.

Context

Status of this document , 4th and 6th paragraphs "A future version of this document will be accompanied by a "Specification Examples & Techniques" document"

Comment

Wrong document name: "Specification Examples & Techniques" . This is also direct duplication with 4th paragraph: "This part of the Framework document family will eventually have an informative accompanying QA Framework: Test Examples and Techniques document"

Proposal

Drop 4th paragraph and fix 6th paragraph to use "Test Examples & Techniques"

Note TGL-JT1.

Context

General

Comment

The style of the language is very poor: the document has the appearance of having been written hurriedly and never reviewed (perhaps not what we would expect from the QA group!). In some cases, the language is so garbled as to render sections ambiguous. Recommend extensive editorial review and enhancement. In particular: Checkpoint 3.1, final 2 paragraphs; Checkpoint 4.2; Checkpoint 4.3.

Proposal

Improve language.

Note TGL-LA2.

Context

1. Introduction

Comment

The Introduction doesn't follow recommendations of SpecGL (checkpoints SpecGL 1.1, 1.2,2.1) and doesn't describe the scope of the specification and classes of product.

Proposal

Follow the structure of introduction used in SpecGL and OperGL and clearly describe the scope and class of products.

Note TGL-JT2.

Context

1. Introduction

Comment

The terms "conformance" and "compliance" are interchanged in an apparently haphazard fashion. The term "compliance" occurs nowhere else in the Framework.

Proposal

Replace all occurrences of "compliance" with "conformance".

Note TGL-JT3.

Context

1.1 Motivation for this guidelines document. 1st bullet.

Comment

First item in bulleted list: it is not realistic to assume that it is possible to eliminate the possibility of misinterpretation of a specification.

Proposal

Substitute "minimized" for "eliminated".

Note TGL-LA3.

Context

1.1 Motivation for this guidelines document. 2nd bullet. "Developing an open conformance test suite for a specification, applicable by any implementation. "

Comment

It's not clear what "open" means here. Availability and licensing policy is in the scope of Operation Guidelines and should be addressed there.

Proposal

Drop the word "open" here and say something about "openness" in Operation Guidelines.

Note TGL-LA4.

Context

1.1 Motivation for this guidelines document. 3rd bullet. "Ensuring the quality of the particular implementation by testing against specifications and conducting interoperability testing with other available implementations."

Comment

There are multiple issues in this sentence. First, a quality of an implementation is much wider notion that just conformance. Along with conformance criteria it may also include performance, reliability, interoperability and other criteria. Also interoperability is a different notion that conformance to the specification. For example, two implementations can be interoperable but non-conformant. And two conformant implementations can be non-interoperable.

Proposal

Change to: "Improving the quality of the particular implementation by testing against specifications and conducting interoperability testing with other available implementations."

Note TGL-LA5.

Context

1.1 Motivation for this guidelines document. "A free-to-use conformance test suite that covers most if not all of the specification requirements, is developed by interested parties across industry, and is applicable to any of the specification's implementations, provides:"

Comment

A freedom to use and a parties involved in the development are in the scope of Operation Guidelines. These are not really relevant for the subsequent logic.

Proposal

To change to: "A conformance test suite that covers most if not all of the specification requirements, and is applicable to any of the specification's implementations, provides:"

Note TGL-LA6.

Context

1.2 Navigating through this document

Comment

What Guideline covers test quality and especially portability?

Proposal

Add guidelines ensuring test quality and portability between implementations.

Note TGL-LA7.

Context

Checkpoint 1.1. Identify the target set of specifications that are being tested "[EX-TECH] For example, XML test suite [@@Link] may not include tests that specifically test the URN format, but XSLT [@@Link] and XQuery [@@Link] test suites will include many tests for XPath functions."

Comment

Is this URN or URI?

Proposal

Change to URI?

Note TGL-LA8.

Context

Checkpoint 1.5. Identify all the discretionary choices defined in the specification [Priority 1]

Comment

Is this really Priority 1? Other similar checkpoints have priority 2.

Proposal

Change priority to 2?

Note TGL-JT4.

Context

Section 1.5.

Comment

Final definition, "Results Verification": presumably the intention is to determine whether an implementation passes or fails a test, not "if a test passes or fails". See also Checkpoint 4.11.

Proposal

Clarification is needed.

Note TGL-LA9.

Context

Checkpoint 2.2. Provide mapping between the test suite structure and the specification structure. [Priority 1]

Comment

All test coverage related items are very important and should have the same priority.

Proposal

Change priority to 2?

Note TGL-LA10.

Context

Checkpoint 3.2. Identify publicly available testing techniques. List the publicly available testing techniques that have been reused [Priority 1]

Comment

Sharing testing techniques is important for test development performance but is not probably P1.

Proposal

Change priority to 2 or 3?

Note TGL-LA11.

Context

Checkpoint 4.1. List available test frameworks and applicable automation. Identify available test frameworks used. If none, justify why new frameworks are needed and existing ones could not be used. [Priority 1]

Comment

This is again related only to performance and efficiency of test development so can be of lower priority. Besides researching all frameworks available in the world could take a lot of unnecessary efforts. Besides adaptation efforts should be taken into account. Also it would be good to choose such a test framework that could be used with different test execution automation systems

Proposal

Change priority to 2 or 3? Limit frameworks to already used in W3C?

Note TGL-LA12.

Context

Checkpoint 4.2. Ensure the framework and automation are platform independent. Demonstrate on 3 platforms. Ensure that the framework and automation are built using open standards. [Priority 1]

Comment

These are maybe two separate checkpoints. Second one about open is not very clear and need clarifications. What those "open standard" means? For example is perl an open standard? python? tcl?

Proposal

Either drop or clarify the second sentence.

Note TGL-JT5.

Context

Checkpoint 4.4.

Comment

It is not reasonable a priori to claim that a TS will "eventually cover all areas of the specification".

Proposal

Remove this sentence.

Note TGL-LA13.

Context

Checkpoint 4.5. Ensure the ease of use for the test automation. Document how the test automation is used. [Priority 1]

Comment

Of course some minimum documentation is required. High quality documentation however requires significant resources and maybe of lower priority. If unqualified it's not clear how to check "ease of use".

Proposal

Split to several checkpoints of different priorities?

Note TGL-LA14.

Context

Checkpoint 4.6. Ensure the framework allows for specification versioning and errata levels. Explain how specification versioning and errata levels are accommodated by the test framework [Priority 2]

Comment

Unsynchronized levels of errata used in specs and test suites is an easy and most common way to non-conformant and non-interoperable implementations. This is absolutely necessary to indicate which version of spec and errata level the tests corresponds. Priority 2 is not sufficient to enforce this.

Proposal

Change priority to 1.

Note TGL-JT6.

Context

Checkpoint 4.11.

Comment

Why is there a requirement to demonstrate results verification by testing three products?

Proposal

Better explain why verification by testing three products is important.

Note TGL-LA15.

Context

Checkpoint 5.2. Ensure the ease of use for results reporting. Demonstrate that the results reporting has sorting and filtering capabilities. [Priority 1]

Comment

Result reporting is important but does it really requires sorting and filtering? It is more important provide a report format that is easily imported into widely available tools (spread sheets, databases, etc)

Proposal

Change priority to 3.

QA Framework: Specification Guidelines
W3C Working Draft 10 February 2003
http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/

Note SGL-LA1.

Context

Status of this document
This part of the Framework document family has a companion QA Framework: Operational Examples & Techniques.

Comment

Just a typo.

Proposal

Change: "Operational Examples" to "Specification Examples"

Note SGL-JT1.

Context

general

Comment

Many if not most of the "Conformance requirements" clauses addressing options in the spec (e.g. Checkpoint 2.4: "If there are several classes of product..." end with a sentence like "It is not applicable if...".

Proposal

Remove all these latter sentences: they are redundant, because the checkpoint already indicates that it is of a conditional nature, and confusing (what is "it"?).

Note SGL-JT2.

Context

Checkpoint 2.1

Comment

In a technological environment in which new innovations and unanticipated uses for existing technology come to light constantly, is it reasonable to expect a spec to identify *all* classes of product it addresses? Example: the XML Forms WG found out that part of their spec was being used in photocopier displays, a use they never imagined at the outset. Another example: the document itself gives only a "non-exhaustive list of classes of products for W3C specifications" in the introductory text for Guideline 2!

Proposal

Remove "all".

Note SGL-JT3.

Context

Checkpoint 8.5

Comment

The checkpoint relates to the use of discretionary items, but its conformance clause refers to the use of profiles.

Proposal

Fix needed.

Note SGL-JT4.

Context

Guideline 9.

Comment

Title reads "Allow extensions or NOT!". What is the purpose of the capitalization and exclamation point?

Proposal

Remove the capitalization and exclamation point.

Note SGL-LA2.

Context

Checkpoint 9.1. Indicate if the specification is extensible.
if the specification writer wants to enhance interoperability by constraining implementer extensions, wording in the specification must indicate this.

Comment

Just a typo.

Proposal

Start with upper case "I": "If the specification..."

Note SGL-LA3.

Context

Checkpoint 10.2. Make normative reference to specifications on which the current specification depends [Priority 2]

Comment

This is critical to define what is normative and what is not. Priority could be higher.

Proposal

Change the priority to 1.

Note SGL-JT5.

Context

Checkpoint 13.3.

Comment

No rationale or discussion is given.

Proposal

Add rationale.

Note SGL-LA4.

Context

Guideline 14. Provide test assertions.
A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behavior.

Comment

It would be good to use consistent definition of an assertion in all QA Framework documents. A definition in Test Guidelines reads: "Test Assertion - A set of premises that are known to be true by definition in the spec." The definition in Specification Guidelines may cause some questions. Is this really mandatory for an assertion to be "independent, complete, testable". An assertion can't be completely independent of others. There can be assertion which are difficult to tests i.e. which can be treated as untestable. Also this should be reworded: "Multiple test assertions can be combined to form a test case" - test case can check several assertion but it is not combined from them.

Proposal

Reword as something like this: A test assertion is a set of premises that are known to be true by definition in the spec. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an statement for requirements in the specification which can be tested independently. One or more test cases can be developed for each test assertion. Also checks for multiple test assertions can be combined to form a test case, in this case a test checks multiple facets of a particular behavior.