Final Minutes - QA Working Group Teleconference 04-April-2005

QA Working Group Teleconference
Wednesday, 04-April-2005
Scribe: Richard Kennedy

(TB) Tim Boland (NIST)
(PC) Patrick Curran (Sun Microsystems)
(DD) Dimitris Dimitriadis (Ontologicon)
(DH) Dominique HazaŽl-Massieux (W3C)
(LH) Lofton Henderson (CGMO)
(RK) Richard Kennedy (Boeing)
(LR) Lynne Rosenthal (NIST)
(DM) Dave Marston (Guest)

(KD) Karl Dubost (W3C, Chair)
(MS) Mark Skall (NIST)


Summary of New Action Items:
AI-20050404-01  (LH) Email QAWG to establish consensus for Dublin F2F dates                       Done
AI-20050404-02  (PC) Publish "Test Development FAQ" to W3C QAWG web site                          Done
AI-20050404-03  (PC) Remove "Working Draft" language from Test Development FAQ (TDFAQ)            Done
AI-20050404-04  (PC) Define "living document" in TDFAQ Introduction                               Done
AI-20050404-05  (DH) Define versioning for TDFAQ                                                  2005-04-11
AI-20050404-06  (PC) Change all "see here" hyperlink descriptions in TDFAQ                        Done
AI-20050404-07  (PC) Incorporate additional OWL reference about their test review process         Done
AI-20050404-08  (PC) Open Bugzilla issue re expanding on the test reuse in a separate document    2005-04-11
AI-20050404-09  (PC) Open Bugzilla issue re expanding coverage metrics discussion in separate doc 2005-04-11
AI-20050404-10  (PC) Develop separate question for using other Working Group's tests              Done
AI-20050404-11  (DH) Contact Voice Browser WG about the Test Case Tracking System                 2005-04-11
AI-20050404-12  (PC) Define "results" in TDFAQ Question 6                                         2005-04-11
AI-20050404-13  (PC) Add language on referencing other files to TDFAQ Question 6                  2005-04-11
AI-20050404-14  (PC) Open Bugzilla issue re expanding this issue in future rev of this document   2005-04-11
AI-20050404-15  (LR/DD) Provide examples of legal problems for TDFAQ Question 7                   2005-04-11
AI-20050404-16  (PC) Add language on the importance of legal issues to TDFAQ Question 7           Done
AI-20050404-17  (PC) Add language on separating test development & execution to TDFAQ Question 8  Done
AI-20050404-18  (PC) TDFAQ Question 9 missing                                                     Done
AI-20050404-19  (LR) Provide XML Core Test Suites examples for TDFAQ Question 11                  Done
AI-20050404-20  (DD) Rewrite 1st sentence of TDFAQ Question 12                                    2005-04-11
AI-20050404-21  (PC) Review remaining TDFAQ Questions & request examples if appropriate           2005-04-11

Added to the minutes at PC's request. "Since I'm updating the document 
on the web, and am renumbering it (to fill in the missing question, and 
also to split a couple of questions that are getting too large), it 
will be a little difficult to map the numbers in the AIs to the current 
version of the document. Would you please insert the following list 
close to the AIs?


 1. What kind of testing is important in the W3C?
 2. When should test development start?
 3. Who will develop the tests?
 4. How do we decide what tests to develop
 5. How many tests are enough?
 6. How should tests report their results?
 7. Do I really have to worry about all that legal stuff?
 8. How should I package and publish my tests?
10. Should I automate test execution?
11. Once I publish my tests, I'm done, right?
12. How should I handle bugs in my test suite?
13. Should test results be published?
14. Should we implement a branding or certification program?"

Previous Telcon Minutes:


Next Meeting: April 11  - The topic is the SpecGL.
RK sends regrets for April 11.

Dublin F2F
Prior date consensus was August 9-11 (Tuesday-Thursday). LH suggested 
August 8-10 (Monday-Wednesday) because those coming from North America, 
who do not have a nice corporate/government travel package, often need 
a Saturday night stay to get reasonable fares. Plus, August is still 
high travel season and he is finding that traveling for a 
Monday-Wednesday meeting improves the choices and fares available. LH 
will email the QAWG for consensus on the F2F dates. Exact meeting 
location and recommended hotels were not discussed.

Test Development FAQ discussion (Lead by PC)
Latest draft:

We are getting very close to publication especially since this is a 
living document. The FAQ's format will be that of a Note on the QAWG 
web page. PC will publish the current version on the QAWG web site 

Labeling the document as a "working draft" implies it is a 
Specification rather than a Note. PC will remove this text.

There was some general discussion on the audience for this note; it is 
W3C or outside persons?

PC will add additional language defining a "living document."

DM expressed concern on how others will link to this document. How will 
the different versions be identified? DH will discuss with DM and pass 
on recommended method to PC.

Question 1: What kinds of testing is important in the W3C?
PC will add descriptive text to all hyperlinks with text like "see 
here" to prevent accessibility issues.

Question 2: When should test development start?
PC will add language stating the importance of starting test 
development early, especially before the Specification is frozen. The 
OWL reference also needs to be updated.

Question 3: Who will develop the tests?
The desirability in including tests developed by other Working Groups 
was discussed. New Specification are increasing modular making their 
reuse easier. PC will add a few sentences discussing this. DH will look 
at making this a separate question after the first publication of this 

Using XForms as an example may be too cutting edge.

The Voice Browser Working Group may have a test case tracking system. 
We need to track this system down.

Question 4: How do we decide what tests to develop?
PC still needs additional example for this question. He mentioned that 
at a recent meeting at Sun, the XQuery Working Group mentioned their 
separate test task force. However, they have not made any solicitation 
for tests from outside of their Working Group.

Question 5: How many tests are enough?
In the current examples nobody is publishing numbers or measurements. 
PC mentioned that test suites need to be versioned and associated with 
the corresponding version of the Specification. DH stated that coverage 
numbers may be too technical for inclusion in this document. PC will 
add some language illustrating some techniques for coverage 

TB supplied some examples:
HTML WCAG2.0 Test Suite (sorted by Guideline)  - gives some indication of test coverage by Guideline and level within Guideline

Example for "reference implementation" for testing: MathML2.0 Test Suite - gives 
"reference" renderings to compare against the renderings under test for 
each test in test suite.

Question 6: How should tests report their results?
DH mentioned that the five test-results states are very useful. PC 
requested additional examples.

The term "results" can mean the output of administering a test or the 
pass/fail outcome of a test. PC will define this in the document.

PC will add a sentence describing reference files and what actual files 
are compared. DH will add this as a separate question after 

Question 7: Do I really have to worry about all that legal stuff? LR 
and DD will provide some examples of legal problems Working Groups have 
had with their Specifications. DH will all language to the question 
stressing the importance of legal issues.

Question 8: How should I package and publish my tests?
PC can not find anyone currently doing this. The SVG Working Group 
package does not have any documentation in its zipped test suites. The 
answer should stress that there are different audiences. A document on 
how to execute tests should be kept separate from a documents 
describing how to develop tests. PC will add appropriate language.

Question 9 (Missing)
PC to renumber document questions.

Question 10: Should I automate test execution?
More examples are needed. The SVG Working Group might have something 
useful as they provide harnesses in their test suites as does the DOM 
Working Group. When XQuery's work is made public, they will also have 

Question 11: Once I publish my tests, I'm done, right?
More examples are needed and the text in red will be stricken. LR will 
provide a link to reflect the XML Core Test Suites that have been both 
augmented and corrected.

Question 12: How should I handle bugs in my test suite?
DD will rewrite the first sentence.

At this point our telcon ended. PC will review the three remaining 
questions and look for examples. The overall consensus is that we are very close to publishing this document.

Received on Monday, 18 April 2005 16:51:59 UTC