QA Working Group F2F Minutes Tuesday, 16-June-2004 - AM -- Scribe: Patrick Curran Attendees: (PC) Patrick Curran (Sun Microsystems) (KD) Karl Dubost (W3C, WG co-chair) (DH) Dominique Hazaël-Massieux (W3C) (LH) Lofton Henderson (CGMO - WG co-chair) (LR) Lynne Rosenthal (NIST - IG co-chair) (MS) Mark Skall (NIST) (DD) Dimitris Dimitriadis (Ontologicon) Regrets: (AT) Andrew Thackrah (Open Group) (VV) Vanitha Venkatraman (Sun Microsystems) Absent: Summary of New Action Items: AI-20040616-1 AI-20040616-2 AI-20040616-3 Agenda: Minutes: o (9:00) TestGL discussion Recap of yesterday's discussions on resource constraints. DD: On the Wiki we might lose control of the document - it could go in the wrong direction. What time-span should we consider? LH: We're not "moving the document to the Wiki". We are taking the main sections/ideas and making Wiki pages that anyone can contribute to. When we pick it up again we - the WG - will decide what we take from the Wiki and incorporate into the document. DD: do the current editors still "own" the document? DH: yes - editors should feel free to work on it. However, the WG will not have the resources to do so. DD: agrees DD: should we still discuss the document during this meeting? PC: yes LR: yes - plus we could figure out how to break it up for the Wiki LH: how about taking all of today for SpecGL, and postponing TestGL until tomorrow morning? LR: no need - the allocated two half-days should be sufficient - continue with TestGL for now DH: we should spend the time to discuss and evaluate the plan to move to the Wiki PC: OK - we will discuss the latest draft, and figure out what [..LH takes over minutes for rest of morning..] DD. (explaining new @@TestGL draft@@) ExTech material will go in here (like SpecGL). PC. Suggests a wiki topic -- enumeration of existing test suites in W3C. DH. QA Matrix has that. DD. Current practices also (i.e., the test practices survey). [...walking thru document...] DD. SoTD is same as SpecGL w/ name changed. Intro, Scope, Goals, Intended audience are pretty general now and not specific to TestGL. DD. Principles: has same Principles as 2 weeks ago draft, in QAH/SpecGL style. Principles at high level, followed by series of "Requirements". PC. Coverage information (Rqt) could be a topic. LH. Would it be better to look at outline view, if our goal is to partition TestGL content into a collection of Wiki topics? [...switching to April 5 outline view...] MS. What do you mean by coverage information? PC. E.g., could indicate on enumerated TAs, which ones have associated TCs. KD. Can you still have a coverage map if don't have TAs? DD. Could have features-to-TCs map, if no TAs. KD. CR requires implementations of each feature in a spec. What's a feature? LH. In SVG, it is a sub-section of the spec. PC. In Java, we do it by TAs. But a subjective measure "by feature" [...minuter lost the point here...]. PC. In Java, you must publish a coverage map by TA, if you propose for example a bit Java technology -- deliver a spec, a TS, and a coverage map, and a "sufficiency claim" for the TS coverage. LR. Based on extensiveness of this discussion, there should be a wiki topic on "coverage". Also need a definition in the glossary (TestGL glossary). MS. Is there a more specific term we can use than "coverage"? Like "requirements coverage"? LR. Shouldn't limit it to assertions coverage. It is "specification coverage", which might be realized as TA coverage, functionality/feature coverage, issue coverage. PC. Ultimately it maps onto a section of the spec. LH. Not necessarily in the case of Web Ontology's model. KD. [...missed his point...] DH. Questions whether it is good that a test might not be traceable back to spec. LR. Suggest again to close this discussion and start wiki topic. LH. Reminder that WebOnt stuff is test-driven spec dev't, and the scope of TestGL is Conf. TSs. So ... interesting to mention but out of scope for TestGL rqts. PC. Sent URL of a public page illustrating a Java coverage statement. [...switching to 20040605 TestGL draft...] DD. Rqt 2.1 -- "Define the contents of the test suite" (within Principle #2 now.) [..goes through listed components..] DD. Rqt 2.2 -- "Specify what tests are to be run". [..explains..] LH. Question about 2.1, and relationship to Test Case Model of KD, and wiki topics. DH. Two topics exist now: test suite high level contents, and detailed metadata. PC. Jumping ahead (to 2.3), "test harness" might be an interesting wiki topic. Also not sure the rqts of 2.3 are testable. DH. Not sure what you mean by "discussion of test harness" as wiki topic. What would be the focus? PC. I think t.h. are good, but not as common as they ought to be in W3C world. DH. Why do we think they're good [..what's the potential wiki topic here..]? PC. [..discussion of the value of a harness..] KD. One common harness for W3C? Or each technology makes its own? Difficult to provide single common harness because of diversity of technology. MS. Mandate the existence of one, but don't try to provide or enforce a single common one. PC. if you don't mandate its use as well, test results will be unreliable (not repeatable/reproducible) for conformance purposes. DD. Must be possible to validate that test results are correct. [..discussion about whether we're going to spawn any wiki topic here... Agreed that Dom will use his discretion based on these discussions for initial selection of topics..] PC. Doesn't hurt to add more wiki topics. LH. Except if get too many and it gets too cluttered, distracting. DD. [..moving on..] Rqt 2.4. [..PC/DD discussion of test results reporting..raw information that allows user to determine, versus the test itself determining pass/fail status..] LR. PC's view of "test reports status" doesn't always work, e.g., in graphics tests where you have to compare two images. DH. Difference is in what you consider to be included as part of the test itself -- instructions to "compare results ... do they match? [then pass/fail report]" could either be considered part of the test or not. [...break...]