WebOnt QAF Operational Guidelines Case Study

In answer to QA wg request, seen in 
http://lists.w3.org/Archives/Public/www-webont-wg/2003Sep/0076.html.  
The Web Ontology Working Group has produced the following case study.


*****

QAF-OPS Case Study for OWL

The following is a case study documenting the quality assurance
activities undertaken by the Web Ontology working group during
development of the OWL language. It is structured as a conformance
evaluation of WebOnt QA activities as prescribed by the CR version of
the QA Framework: Operational Guidelines [QAF-OPS].  We have included
within this text the Checkpoints, Conformance Requirements and
occasional Guidelines from QAF-OPS as context for readers from WebOnt.
Text documenting WebOnt actions is prepended with "WebOnt:".

===

Checkpoint 1.1. Define QA commitment levels for operations,
specifications, and test materials. [Priority 1]

Conformance requirements:
the WG MUST define its commitment level to QA Framework: Operational
Guidelines (this specification) -- A, AA, or AAA;

for any Recommendations that it plans to produce, the WG MUST define
its commitment level to QA Framework: Specification Guidelines -- A,
AA, or AAA;

for any Test Materials that it plans to produce or adopt, the WG MUST
define its commitment level to QA Framework: Test Guidelines -- A, AA,
or AAA.

A new or rechartering Working Group MUST define its QA commitment
level in its Charter; 

an existing Working Group MUST document its QA commitment level in
some consensus record.

WebOnt: 
No commitment has been made to QAF by WebOnt.  The WG has looked at
the QAF-Specification Guidelines with mixed opinion.  The QA Test
Guidelines have yet to be reviewed by WebOnt.  This text represents
the WGs initial response to the QA Ops Guidelines and any WG decision
about commitment would occur subsequent to completion and working
group consideration of this response.


Checkpoint 1.2. Commit to test materials. [Priority 2]

Conformance requirements:

the WG MUST commit to produce or adopt at least some test materials
for each of the WG's specifications before it becomes Recommendation;

a new or rechartering Working Group MUST define its test materials
commitment in its Charter;

an existing Working Group MUST document its test materials commitment
in some consensus record.

WebOnt:

WebOnt did not make any commitment to test materials in its charter.
Committed resources to [TEST] document when TEST taskforce was formed
(see consensus record of Feb 28 2002 telecon
http://lists.w3.org/Archives/Public/www-webont-wg/2002Mar/0029.html)

The test case document became a Working Draft component of the OWL
specification when published on 24 October 2002.  Subsequent versions
of the Test document were part of LC and CR specifications and key in
the exit criteria for CR.


Checkpoint 1.3. Commit to complete test materials. [Priority 3]

Conformance requirements:

the WG MUST commit to produce or adopt a complete test materials
before Recommendation, where complete is defined as: at least one test
case for every identifiable conformance requirement of the
specification;

a new or rechartering Working Group MUST define its test materials
commitment in its Charter;

an existing Working Group MUST document its test materials commitment
in some consensus record.

WebOnt: 

Conformance classes were defined and reflected in the OWL Test
document.  However, the tests were not complete in regard to testing
all the features of the language or all its envisioned usages.  It was
felt that it was impossible to know what the complete list of usage
classes would be for the language and that a complete conformance test
suite would be inappropriate for a technology where considerable
development was still expected.  Instead a commitment was made to
produce an open set of tests which continued to expand and evolve
until PR in order to: 1) Illustrate usages of language features, 2)
illustrate characterizing features of known implementation classes, 3)
test the current boundaries of those implementation classes, 4)
capture and illustrate resolutions of issues with the design of the
language, and 5) clarify user confusion discovered by tool
implementers other researchers.

This position was discussed both in the working meetings of the
group and in its email list and was documented in published drafts
of the Test document as early as October 2002.


Checkpoint 1.4 Enumerate QA deliverables and expected milestones.
[Priority 1]

Conformance requirements:
a new or rechartering Working Group MUST document its QA deliverables
and milestones in its Charter;

an existing Working Group MUST document its QA deliverables and
milestones in some consensus record.

WebOnt: 

The charter for this working group makes no explicit mention of QA
deliverables or milestones.  However, three OWL documents: "OWL Web
Ontology Language Use Cases and Requirements", "OWL Web Ontology
Language Guide", and "OWL Web Ontology Language Test Cases" were
committed to early in the WG's lifecycle.  These documents provide the
QA deliverables: use cases, primer, and a collection of test
assertions respectively.


Checkpoint 1.5. Define QA criteria for Recommendation-track
advancement. [Priority 2]

a new or rechartering Working Group MUST, in its charter, specify the
QA criteria required for transition of the WG's specifications between
major Recommendation-track maturity levels;

an existing Working Group MUST, in some consensus record, specify the
QA criteria required for transition of the WG's specifications between
major Recommendation-track maturity levels.

WebOnt: 
As already stated for checkpoint 1.4, no QA deliverables or milestones
were a part of the charter.  Recommendation-track phase change
criteria were keyed to implementation experience described in terms of
the test cases, as well as, to resolution of public comment on the QA
materials already noted above.  This required completion of all these
materials prior to entering PR.


Guideline 2. Commit to resource level for Working Group QA activities.

Checkpoint 2.1. Address where and how conformance test materials will
be produced.  [Priority 1]

a new or rechartering Working Group MUST, in its charter, address who
will produce its test materials and how;

an existing Working Group MUST, in some consensus record, document who
will produce its test materials and how.

WebOnt:  
What this really means is whether Test Materials (TM) will be produced
within the WG or by some other party (organization or individual). As
already stated WebOnt committed to produce its own test cases.


Checkpoint 2.2. Address QA staffing commitments. [Priority 1]

Conformance Requirements:

a new or rechartering Working Group MUST, in its charter, commit to a
staffing resource level for the tasks necessary to meet its total QA
commitments according to its where-and-how plan;

an existing Working Group MUST, in some consensus record, commit to a
staffing resource level for the tasks necessary to meet its total QA
commitments according to its where-and-how plan.

WebOnt:
Committed resources to Test document when TEST taskforce was formed
(see consensus record of Feb 28 2002 telecon
http://lists.w3.org/Archives/Public/www-webont-wg/2002Mar/0029.html)


Checkpoint 2.3. Request allocation of QA resources to the Working
Group.  [Priority 1]

a new or rechartering Working Group MUST, in its Call for
Participation, request that participating members include QA
specialists in their staff-resource allocation to the WG;

an existing Working Group MAY make an external appeal for QA-specific
resources in one of various other ways.

WebOnt: 
WebOnt did not request QA-specific resources in its call for
participation.  Even so, the group had sufficient resources to prepare
the Test, Guide, and Use Case documents already mentioned.


Guideline 3. Synchronize QA activities with the specification
milestones.

Checkpoint 3.1. Synchronize the publication of QA deliverables and the
specification's drafts. [Priority 2]

the Working Group MUST publish QA deliverables, including at least the
test materials to which the WG has committed, concurrently with each
Working Group specification publication milestone.

WebOnt: 
The test cases were not ready for the OWL Last Call, and were published 
shortly after. This means that one or two tests in the OWL Test Last Call 
reflect last call issue resolutions, rather than the text of the other OWL 
last call documents. However, this did not appear to present any difficulties.
It may have been easier if we had been clearer in our planning and had 
decided earlier that we would do that. The test document was central in the 
Candidate Rec phase, although in practice implementors were encouraged to 
work with the public editors' draft of the test document (particularly the 
list of proposed, approved and obsoleted tests in the test manifest). This 
allowed rapid feedback on WG decisions and new tests. As we approach 
Proposed Recommendation, it is arguable that we would have done better if 
we had planned for a staggered release, with the test document coming last. 
As it is, key conformance clauses are in the test document which prevented 
such staggering.


Checkpoint 3.2. Support specification versioning/errata in QA
deliverables. [Priority 1]

Conformance Requirements:

the Working Group's test materials MUST support the unambiguous
association of test materials versions to specification versions and
errata levels;

specification versions and errata support of the Working Group's test
materials MUST be documented in test materials documentation;

the Working Group SHOULD include specification versioning/errata
considerations in any other QA deliverables such as intermediate
planning documents.

WebOnt: 
This checkpoint is applicable only to post recommendation
specifications which do not yet exist for OWL.  That scope is not
clear from the checkpoint or its conformance requirements.


Guideline 4. Define the QA process.

Checkpoint 4.1. Appoint a QA moderator. [Priority 1]

Conformance Requirements:

the Working Group MUST identify a person to manage the WG's quality
practices.

WebOnt:
To the extent that a QA moderator would be strictly concerned with TM
development, the editor and co-editor of the Test document have
fulfilled that role in WebOnt.  


Checkpoint 4.2. Appoint a QA task force. [Priority 2]

Conformance Requirements:

the Working Group MUST identify and assign a QA task force for the
tasks necessary to meet the QA commitment level and the committed QA
deliverables, as identified in checkpoints 1.1 - 1.5.

WebOnt: 
A Test taskforce was formed but most test discussions were conducted
in the working group as a whole and test material contributors were
not confined to those officially belonging to the Test taskforce.  All
taskforces within webont were treated more like roles taken on by the
whole group for addressing specific aspects of specification
development, rather than subsets of the group primarily focused on
those aspects.


Checkpoint 4.3 Produce the QA Process Document [Priority 1]

Conformance requirements:

the Working Group MUST produce a QA Process Document; 

the Working Group's QA Process Document MUST be publicly readable; 

the Working Group's QA Process Document MUST address at least all of
the topics required of it by other checkpoints in these operational
guidelines.

WebOnt: 
WebOnt did not produce a QA process document, per se.  The test
document outlined the character, structure, and form of the test
materials, as well as, explaining who and how tests were created,
approved and modified.  There is no evidence that the absence of a
dedicated QA process document hampered the development of OWL test
material or other quality assurance related materials.


Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]

Conformance requirements:

the Working Group's QA Process Document MUST specify at least one
public archived mailing list for QA announcements and submission of
public QA comments; 

the Working Group's QA Process Document MUST establish a publicly
readable "Test" Web page.

WebOnt: 
The Test document specifies public-webont-comments@w3.org as the
appropriate place for submission of implementation experience.
Significant discussion on test cases has taken place there,
particularly as OWL nears PR.  Most of the test materials are packaged
within the Test document.  However, a web repository is also provided
for accessing test cases.


Checkpoint 4.5. Define branding policy details. [Priority 3]

Conformance Requirements:
the WG MUST document in its QA Process Document the branding policy
details, if branding is to be supported.

WebOnt: 
WebOnt does not currently sanction any OWL conformance claims,
therefore this checkpoint is not applicable.


Guideline 5. Plan test materials development. [Priority 2]

Conformance requirements:

the Working Group's QA Process Document MUST define a framework for
test materials development, that at least describes how to develop,
document and use the tests.

WebOnt: 
(based on a literal reading of the above requirement) As previously
mentioned, Appendix A of the Test document describes how tests can be
created and submitted.


Checkpoint 5.2. Ensure test materials are documented and usable for
their intended purposes. [Priority 1]

Conformance requirements:

the Working Group MUST have user documentation for its test materials
that instructs the use of the test materials for the full range of
their intended purposes.

Webont: 
OWL tests are embedded in a document which explains the test
type, class and purpose. Manifest files provide these same descriptions in
the form of machine readable metadata.  Implementation testing using
these test cases was conducted by many who were not party to the
development of the test materials or any other working group
activities, yet were able to run the tests and provide machine
readable result files.


Checkpoint 5.3. Define a contribution process. [Priority 2]
Conformance requirements:

the Working Group MUST describe in its QA Process Document where, how,
by whom, and to whom test materials submissions are to be made.

WebOnt: This is all outlined in the Creation portion of Appendix A of
the Test document.


Checkpoint 5.4. Address license terms for submitted test
materials. [Priority 1]

Conformance requirements:

in its QA Process Document the Working Group MUST define a submission
license policy applicable to test materials submitted to the WG by
external parties;

the Working Group's submission license policy MUST include at least an
outline of terms, conditions, constraints, and principles that will
govern acceptable submissions to the WG.

WebOnt: 
Test contributions were restricted to those within the working group.
The only licensing terms were those normally associated with W3C
working drafts.  The licensing boilerplate suggested in the test
document's Stylistic Preferences appendix implied W3C copyright for
all tests.

One potential test contribution appeared to be discouraged by this (the 
handing of copyright to W3C). Offlist, the editor encouraged the submitter 
to suggest different terms, but that appears to have presented a hurdle, 
and no contribution or further discussion of IPR terms was had.


Checkpoint 5.5. Define review procedures for submitted test
materials. [Priority 2]

Conformance requirements:

in its QA Process Document, the Working Group MUST define a procedure
for reviewing test materials contributions; 

the Working Group's procedure for reviewing test materials
contributions MUST at least address criteria for accuracy, scope, and
clarity of the tests.

WebOnt: 
Test: Appendix A describes the process for WG Approval of tests.
Implied by this is review of the test, but no explicit process or
criteria are specified for use in such a review.  During CR the WG
tended to accept test passes by two independent systems as sufficient
evidence for approving a test (in the absence of opposition).


Guideline 6. Plan test materials publication.

WebOnt discussion. The handling of web publication and access test
materials for webont is very similar to the approach taken by the SVG
project.  A W3C hosted CVS repository is used to manage the material.
Access is via website or test document.  A tabular view of the
submitted implementation test results is also available.


Checkpoint 6.1. Ensure a suitable repository location for test
materials. [Priority 1]

Conformance requirements:

the Working Group MUST keep its test materials in repository locations
that are secure, reliable, and freely accessible.

WebOnt: 
WebOnt test materials are kept in a CVS repository hosted by the W3C.


Checkpoint 6.2. Define the licenses applicable to published test
materials. [Priority 1]

Conformance requirements:

the Working Group MUST, in its QA Process Document, define the
licenses that are applicable to published test materials.

WebOnt: 
WebOnt tests are part of the Test document which clearly indicates in
its title block that W3C software licensing rules apply.


Checkpoint 6.3. Describe how and where the test materials will be published. 
[Priority 2]

Conformance requirements:

in its QA Process Document the Working Group MUST document the planned
Web location and method for publication of its test materials.

WebOnt: The OWL Test document states where and how the tests are
published on the web.


Checkpoint 6.4. Provide a conformance verification disclaimer with the
test materials. [Priority 1]

WebOnt: 
The OWL Test document includes no conformance verification disclaimer.
This could be because it already asserts that the tests it contains do
not constitute an OWL conformance test suite.


Checkpoint 6.5. Promote testing and the publication of test results. 
[Priority 2]

Conformance requirements:

in its QA Process Document the Working Group MUST document a plan to
engage implementors to participate in conformance testing activities;

in its QA Process Document the Working Group MUST document a plan to
encourage the publication of test results, including sample scenarios
of where and how such publication can be done;

in its QA Process Document the Working Group MAY identify a
WG-sponsored Web site for publishing collected results or a directory
of results.

WebOnt: 
WebOnt has no QA Process Document, per se.  The OWL Test document
which contains material similar to that anticipated for a Process
Document, does not contain plans outlining how to involve implementors
in conformance testing activities for OWL, nor a plans for collecting
and publishing the results.  Even so, implementors did do a great deal
of testing, submitting the results to the group.  WebOnt makes these
results available through an OWL Test Results page that displays them
by test category, by test, and by implementation.


Guideline 7. Plan the transfer of test materials to W3C if needed.

WebOnt: WebOnt developed all its test materials within the group,
rendering most of the checkpoints for this Guideline not applicable.


Checkpoint 7.1. Perform a quality assessment of any test materials
that are candidates for transfer.  [Priority 2]

Conformance requirements:

as a part of any test materials transfer process, the Working Group
MUST perform and record an assessment of the quality of the test
materials.

Webont: No transfer needed. Not applicable.

Checkpoint 7.2. Identify sufficient staff resources to meet the needs
of any transferred test materials. [Priority 1]

Conformance requirements:

as a part of any test materials transfer process, in some consensus
document the Working Group MUST identify and assign staff resources
for the tasks associated with ongoing test materials development and
maintenance after the transfer.

Webont: 
The WG members submitting the tests did the bulk of the work required
to transform the tests from their original form to that appropriate
for the OWL Test suite. Further work was done by the editors.


Checkpoint 7.3. For any transferred test materials, resolve all IPR
issues with the external party that produced the test materials.

Conformance requirements:

as a part of any test materials transfer process, the Working Group
MUST have a documented agreement with the external entity that covers
the IPR aspects that are applicable to the transferred materials.

Webont: Not applicable.


Guideline 8. Plan for test materials maintenance.


WebOnt discussion: The Test cases have been made a part of the OWL
specification and are expected to be maintained along with the
specification.  The working group is expected to close subsequent to
OWL Recommendation and an errata period.  No plans have been made for
supporting Test material maintenance outside of the working group.


Checkpoint 8.1. Provide for the long-term maintenance of the
contribution and review procedures.  [Priority 3]

Conformance requirements:

in some consensus document, the Working Group MUST define a plan and
identify resources for the maintenance of the test materials'
contribution and review procedures throughout the entire life cycle of
the test materials and the Recommendation itself.

WebOnt: No plans have been made for the maintenance of OWL test
materials when WebOnt disbands.


Checkpoint 8.2. Specify a test materials update procedure to track new
specification versions/errata.  [Priority 1]

Conformance requirements:

in its QA Process Document the Working Group MUST specify procedures
to update the test materials to track new specification versions and
errata levels.

WebOnt: WebOnt has not documented any unique procedures for updating
test materials relative to post-Recommendation versions and errata
levels. However, since the test materials are a part of the
specification set for OWL, any published release of OWL should have
consistent test materials by virtue of normal publication procedures.


Checkpoint 8.3. Identify a procedure for test validity appeals.
[Priority 2]

Conformance requirements:

in its QA Process Document the Working Group MUST identify a
communication channel for appeals of test validity and a procedure for
resolving such appeals.

WebOnt: 
The Test document included a process for test vetting and subsequent
decommisioning if appropriate.  During the development of OWL, the
Issue list provided a formal mechanism leading to creation or removal
of tests.  During recommendation process phase-transitions, public
comment procedures were used and new information sometimes led to new
or revisited issues and thus new or revised tests.  These processes
all were designed to work in the context of an active WebOnt wg, and
will not apply after its dissolution.


[QAF-OPS] QA Framework: Operational Guidelines
http://www.w3.org/TR/2003/CR-qaframe-ops-20030912/

[TEST] OWL Web Ontology, Test Cases
http://www.w3.org/TR/2003/PR-owl-test-20031215/

Received on Friday, 19 December 2003 12:36:42 UTC