Attendees:
(TB) Tim Boland (NIST)
(PC) Patrick Curran (Sun Microsystems)
(LH) Lofton Henderson (CGMO) (at :30 past)
(LR) Lynne Rosenthal (NIST)
(DH) Dominique Hazael-Massieux (W3C)
(DD) Dimitris Dimitriadis (Ontologicon)
(RK) Richard Kennedy (Boeing)
(DM) David Marston (guest - IBM Research)
Regrets:
(KD) Karl Dubost (W3C, Chair)
(MS) Mark Skall (NIST)
Absent:
Summary of New Action Items:
AI-20050111-01: LR to send email to list re: proposed telecon time change
AI-20050111-02: DH to send email to list re: Technical Plenary registration
AI-20050111-03: DH to provide example of TCMS?
AI-20050111-04: TB to provide pointer to CSS Selectors Implementation Report Template?
AI-20050111-05: DH to provide XSLT stylesheet used in HTML?
AI-20050111-06: LH to provide pointer to legal issues document?
AI-20050111-07: PC to create issues list related to TEST FAQ?
AI-20050111-08: PC/DH to "formally" publish revised TEST FAQ (with date/version)
in WG space within several weeks
Agenda (Jan 12)
Previous Telcon Minutes (Jan 5)
Minutes:
There was interest in reopening the issue of moving the telecon time to Mondays 11am-12pm, because several members have conflicts, although the Wednesday time slot allows adequate preparation of the agenda and meeting materials during the week before telecon time. ACTION LR will send out an email to this effect to the list for email discussion. This issue will be a topic of discussion on the telecon of the 26th.
Participants are encouraged to make hotel reservations and to register for the conference ASAP; the deadline for getting the most favorable rate at the hotel where the meeting will be held is Feb 5, but the hotel would like to know the numbers much sooner than this. ACTION DH will send a note will be sent to the list to this effect.
Reviewers should have their reviews done by Jan 28
The CSS2.1 exit criteria for CR was discussed as an example of what the CSSWG is (and has been) doing in achieving interoperability by having two separate implementations pass the same tests in the CSS2.1 Test Suite by end of CR period. Five conditions are stated as follows: (1) there must be two interoperable implementations of every feature, (2) a minimum of six months of the CR period must have elapsed, (3) the CR period will be extended if implementations are slow to appear, (4) features that were not in CSS1 will be dropped if two or more interoperable implementations are not found by end of CR period, and (5) features will also be dropped if tests of sufficient quality are not developed by end of CR period. The goal of these conditions is to attempt to ensure that all features remaining in the specification are sufficiently and consistently implemented by multiple offerers.
A detailed review was undertaken of the latest TEST FAQ document, starting from the beginning of the document, after some general comments.
General Comments:
Is the style/level of detail appropriate? It was felt by telecon participants that the style is appropriate and at the right level of detail (informal) (each question about 10-20 lines long, with a few examples to illustrate/augment but not to make the text corresponding to each question too long ). A guideline might be that if a question/answer does not fit on a screen in its entirety it may be too long.
A concern was raised about how other readers would view the document. Readers may be familiar with their technologies but not necessarily familiar with testing issues per se, and jumping immediately into the questions/answers may be too abrupt and assume too much testing knowledge on the reader's part. Consequently, it was decided to insert a short introduction before the first question, possibly dealing with benefits of testing, audience for document, purpose, etc. It was an issue whether benefits of testing should have its own question or be folded into the existing first question.
Another general concern was raised about providing enough examples and pointers to augment the text for each question, so that WGs could consult the document for help without having to do this work themselves. It was emphasized that the commitment of the QAWG is to maintain this document and answer questions relating to the document (provide feedback to readers), so that work is not unnecessarily "pushed off" to the WGs when information pertaining to that work could be maintained in this document.
It was emphasized that it is important to keep the document up-to-date, so that even though the document is a snapshot of state of the art at a certain pont in time, that snapshot will change over time as new material is added. Related to this, an example of a Test Case Management System was discussed for possible inclusion as an example (ACTION DH to try to get one?).
Members were also asked to consider the possibility of template use (issue?).
The question of EARL as a reporting tool was also mentioned.
Another general suggestion was to switch the wording to be somewhat more "positive" in tone where appropriate.
Specific Comments on Each Question/Answer:
For first question ("What kinds of testing are done in the W3C?"), it was determined that it is not necessary to define "test"; readers would understand what a test is about. Added to TEST FAQ will be a note, to the effect that "those who have questions about testing should come here..", as well as a "request for feedback" (form?).
For second question ("When should test development start?"), DH's suggestions on this question (ref also earlier email from DH) were accepted, along with the OWL guidelines as example? Also, LR had a concern about possible confusion involving first and third paragraphs saying two different things?.
For third question ("Who will develop the tests?") , first paragraph, it was suggested that the tone of the language be editorially modified somewhat. It was pointed out that it is possible to acquire tests from other sources, as well as to re-use tests developed by other WGs if possible, and this will be added to the document. DH indicated that at Technical Plenary Day there is scheduled to be a session on sharing testing tools, so WG members might be asked for contributions.
Concerns were also raised on the optionality implied in use of words like "metadata" along with "expected results", so wording may be changed to separate "expected results" from "metadata" in this context. It was pointed out that by default, test cases come with expected results (those results against which actual results are compared to derive "Pass/Fail" outcomes).
As an aside, it was also pointed out that "negative tests" are also important, and so language to this effect will be added. On the last paragraph, it was indicated to address policies about contributions more, and to go further in defining a process for submission, as well as to split the text up into test management policy text and test management system text; re: the former, it may be possible to provide an example re: OASIS TC on XSLT/XPath Conformance once this splitting has been done.
For fourth question ("How do we decide what tests to develop?"), it was pointed out that a TCMS could include a coverage map. Any additional examples that could be provided, as well as any additional criteria to add to the list, may be helpful for this question.
For the fifth question ("How many tests are enough?"), what is a "coverage goal"? The issue of how many tests are enough has been raised in the context of XSLT and XQuery. Another issue is how the director may signal that a CR exit criteria has been met, in terms of number and coverage of tests - what evidence is provided? It was pointed out that the QA Team does review the test evidence prior to making a decision for a specification to exit CR. ACTION TB will forward a pointer to a CSS Selectors Implementation Report Template.
For the sixth question ("How should tests report their results?"), ACTION DH to provide XSLT stylesheet used in HTML(?) for an example supporting this text.
For the seventh question ("Do I really have to worry about all that legal stuff?"), LH has a detailed writeup on legal issues. ACTION LH to provide a pointer to this writeup.
For the eighth question ("How should I package and publish my tests?"), it was pointed out that running XQuery tests, some configuration/ adaptation is required to run the tests, because of different types of servers; this is also an issue with running tests re: Java servers; however, it should always be clearly defined what can be changed about the tests and what cannot.
For the ninth question ("What should the test documentation cover?") , it was suggested that a pointer should be included to the most comprehensively documented test suite. QA also has a test matrix.
For the tenth question ("Should I automate test execution?") , it was suggested that more detail and examples may be needed.
For the eleventh question ("Once I publish my tests, I'm done, right?"), it was asked as to how should versioning of test suites be handled. An answer was that if a specification is revised, then the test suite should certainly be revised, but test suites can also be revised independently of this (for example, additional tests are included, or bugs are fixed).
DM said that while patch/update of test cases is handled acceptably, there are larger issues about errata of either the specification or the tests.
ACTION PC will create an issues list containing issues arising out of document review thus far.
ACTION: PC/DH will "formally" publish an updated version of TEST FAQ (reflecting comments agreed upon thus far (including what DH sent in his earlier email?)) in WG space within the next several weeks.
WG members voiced thanks to PC for his efforts on TEST FAQ.