Reviewed by: Jack Morrison
Date: August 21, 2002
The MathML Working Group has already completed two generations of specification development, so this review involved looking at both working group charters, specifications and test suites. Doing the review in this manner allowed the reviewer to determine what changes had been made over time.
Guideline 1. Integrate Quality Assurance into Working Group activities.
The MathML 1.0 WG Charter makes no mention of QA or testing suite activity as part of the specification development process. The MathML 2.0 WG Charter defines "Maintenance of the Test Suite and Validation services for MathML" as a "Criteria for success". From this it appears that although not specifically mentioned in the 1.0 charter, some level of test suite development was done as part of the 1.0 process. This shows recognition that a basic commitment to QA is important, although not defined at as detailed a level as the Guidelines propose.
1.1 In the Charter, specify the level of commitment to QA. Plan to have at least a commitment to "QA level three". [Priority 1]
No specific commitment to those items defined in Commitment Levels 1-3 is provided in either the MathML 1.0 or 2.0 Charter. However, as noted above, test suite development and maintenance was performed by both working groups.1.2 In the Charter, plan at least for the"QA level five". [Priority 2]
See 1.1.1.3 In the Charter, plan for the"QA level seven". [Priority 3]
See 1.1.1.4 In the Charter, enumerate QA deliverables and expected milestones. [Priority 1]
The MathML 1.0 charter has no milestones associated with it. The MathML 2.0 Charter indicates that one of the deliverables is "an adjustment to the MathML test suite" related to modularization of the MathML 2.0 specification. The milestones include 2 intermediate and 1 final update to the Test Suite.1.5 In the Charter, associate QA criteria with WG milestones. [Priority 2]
Neither the MathML 1.0 or 2.0 Charter defines specific criteria that must be met to achive phase entry or completion milestones. However, from reviewing the MathML 2.0 Implementation and Interoperability Report it appears that a test suite of standard examples was required to exit CR.Guideline 2. Define resources for Working Group QA activities.
Neither charter contains anything related to staffing for QA resource.
2.1 In the Charter, address where and how conformance test materials will be produced. [Priority 1]
See Guideline Summary.2.2 In the Charter, address QA staffing commitments. [Priority 1]
See Guideline Summary.2.3 In the Call for Participation, request allocation of QA resources to the Working Group. [Priority 1]
See Guideline Summary.Guideline 3. Synchronize QA activities with the milestones for WG deliverables.
The plan for synchronization of specifications and test suites is defined in the 2.0 charter. No reference exists in the 1.0 charter.
3.1 Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]
Section 7. Milestones of the MathML 2.0 Charter (http://www.w3.org/Math/Documents/Charter2001.html#milestones) defines both specification and test suite deliverables.3.2 Support specification versioning/errata in the QA deliverables. [Priority 3]
There appears to be no versioning/errata information provided in either test suite. An updated MathML 2.0 Test Suite was produced, but it was primarily a major structuring of 1.0.Guideline 4. Define the QA process.
In reviewing the charters and mail archives for both 1.0 and 2.0, I could not find any specific references to any of the guidelines in this section.
4.1 Appoint a QA moderator. [Priority 1]
I was able to determine that Robert Miner (RobertM@dessci.com) was the primary test suite author for MathML 2.0, but it is unclear if he actually fulfilled the role of a moderator. For MathML 1.0 it is unknown how the process operated.4.2 Appoint a QA task force. [Priority 2]
It does not appear a formal task force was appointed. QA functions were done on an as needed basis by those who volunteered.4.3 Produce the QA Process document. [Priority 1]
No QA Process document exists for MathML 1.0 or 2.0. The lack of this document means a large number of the guidelines were not met.4.4 In the QA Process document, define means for QA-related communication. [Priority 2]
No QA Process document exists.4.5 In the QA Process document, specify the framework for test materials development. [Priority 1]
Although there does not appear to be a specific process document, the test suite was developed using the W3C Math XSL Stylesheet (http://www.w3.org/Math/XSL).4.6 Specify a policy for branding materials. [Priority 3]
See Guideline Summary.Guideline 5. QA Process: Plan test materials development.
It appears that no QA Process document exists for MathML 1.0 or 2.0.
5.1 In the test framework, ensure test materials are documented and usable for the intended purpose. [Priority 1]
The MathML 2.0 Test Suite uses XML as a framework. It is both comprehensive and well layed out. For 2.0 it underwent a major rewrite so that it was organized to follow the structure of the MathML specification. I was not able to find a copy of the 1.0 Test Suite.5.2 In the QA Process document, define a contribution process. [Priority 2]
No QA Process document exists.5.3 In the QA Process document, define the licenses applicable to submitted test materials. [Priority 1]
No QA Process document exists.5.4 In QA Process document, define review procedures for submitted test materials. [Priority 2]
No QA Process document exists.5.5 Test materials review process includes but not limited to checking for accuracy, scope and clarity of the tests. [Priority 1]
Based on the Milestones in the MathML 2.0 Charter and the completeness of the test suite, I believe that this guidelines was met, although I have no documentation to prove it.Guideline 6. QA Process: Plan test materials publication.
Since no QA Process document exists for MathML 1.0 or 2.0 most of these guidelines were not met.
6.1 Ensure a secure, reliable, freely accessible repository location for test materials. [Priority 1]
All test suite materials reside on W3C servres and are accessible form the W3C MathML website.6.2 In the QA Process document, define the licenses applicable to published test materials. [Priority 1]
No QA Process document exists.6.3 In the QA Process document, describe how the test materials will be published and point to the corresponding web page. [Priority 2]
No QA Process document exists.6.4 Provide a disclaimer regarding the use of the test materials for compliance verification. [Priority 1]
No disclaimer currently exists.6.5 In the QA Process document, describe how the test results for the products can be published. [Priority 2]
No QA Process document exists.Guideline 7. Plan the transfer of test materials to W3C if needed.
As far as I can tell all test suite development was done internal to the working group.
7.1 If transfer of the test materials to W3C is planned, perform an assessment of their quality. [Priority 2]
N/A7.2 If transfer of the test materials to W3C is planned, identify sufficient staff resources to meet the requirements. [Priority 1]
N/A7.3 If transfer of the test materials to W3C is planned, resolve IPR questions and reach agreement with the external party that produced test materials. [Priority 1]
N/AGuideline 8. Plan for test materials maintenance.
Since no QA Process document exists for MathML 1.0 or 2.0 most of these guidelines were not met.
8.1 Maintain the contribution and review procedures throughout the entire life cycle of the test materials and the standard itself. [Priority 3]
Since no QA Process document exists for MathML 1.0 or 2.0, it is not possible to determine what the porcess was defined to be.8.2 In the Working Group's QA process document, specify a procedure for updates of the test materials to track new errata/specification versions. [Priority 2]
Again, since no QA Process document exists it is not possible to determine WHAT the process was, but based on the Milestones in the MathML 2.0 Charter it appears that updates were done. Unknown for MathML 1.0.8.3 In the Working Group's QA process document, identify the communication channel and procedure for appeals of tests validity. [Priority 2]
No QA Process document exists.[Optional: opinions, observations, assessment, judgements, etc, about the subject of your review, about the review process, the Guidelines document, etc.]