Re: Fwd: Re: raw draft minutes

Included are the draft minutes for Thursday morning. Noticed that there are two
due dates missing in the action list.

Sandra

QA Working Group Teleconference
Thursday 6-March-2003
--
Scribe: Sandra I. Martinez

Attendees:
  (KD) Karl Dubost (W3C, WG co-chair)
(PF) Peter Fawcett (RealNetworks)
  (DH) Dominique Hazael-Massieux (W3C - Webmaster)
(LH) Lofton Henderson (CGMO - WG co-chair)
(SM) Sandra Martinez (NIST)
  (LR) Lynne Rosenthal (NIST - IG co-chair)
(MS) Mark Skall (NIST)
(OT) Olivier Thereaux
(DD) Daniel Dardailler (W3C - IG co-chair)
(DM)David Marston
(PC) Patrick Curran (Sun Microsystems)



Summary of New Action Items:

A-2003-03-07-1: Action to Peter to send comments to the  form for OpsGL. By 
Tuesday, March 10.
A-2003-03-07-2: Action to Lynne to include Ian Jacob’s comments into the 
form for SpecGL. By Tuesday, March 10.
A-2002-03-07-3: Action to Patrick to review to QAWG charter. By end of next 
week
A-2002-03-07-4: Action to Lofton Updating the QA PD template and message to 
chair list  March 20
A-2002-03-07-5: Action to David to draft resolution to issue 23.
A-2002-03-07-6: Action to Lynne to write checkpoint to reflect resolution 
of   issue 107 march 20
A-2002-03-07-7: Action to Patrick to circulate a set of suggested list of 
attributes that will lead to collapsing the checkpoints in GL 1.
A-2003-03-07-8: Action to Peter to combine GL 2 and 3. By Friday, March 14.
A-2003-03-07-9: Action to Patrick to restructure of the guideline 4.



Peter initiated the meeting providing a brief summary of the results from 
the break out team’s review of the Process document using the process 
document template.  The activity resulted in a set of comments that are 
going to be incorporate in the OpsGL comments form. Patrick took this action.
Lynne will incorporate Ian Jacobs’s comments into the SpecGL comments form.

As a result of this review, Lofton recommended that a review of the charter 
should also be done. Patrick took the action of performing the review.

TestGL issues resolution

Issue 13: Is inter-standard or multi-standard interoperability within the 
QA scope? Issue raised in email thread (DM, 2001-11-07, 2nd pgph).
         MS: There are no requirements for interoperability. We need 
specific checkpoints.
LR: The requirements for interoperability are being addressed in other 
specifications. If they are represented in those specifications they are 
going to be tested.

         LF: This is an old issue, just noticed the e-mail that was generated.

         DM: The intention of this issue was related to the pipeline.

         MS: There is a problem with the way we are capturing the issues.
LR: This type of information should not be included in the TestGl at this 
point. We need to keep this document simple.

KD  Agree. The TestGl needs to be simple and this will not prevent writing 
a separate document to address clarification items.

PC  Also agree to keep the document simple. Put some words in the document 
about extensibility/optionality. Propose to address this in the SpecGl, not 
in TestGl.

LH : Propose to cut and paste clarification into the body of the 
issue.  Also propose to go with Lynne’s proposal.
The issue is closed.

Issue 79: Should Test Guidelines address how to write a good test suite?

         LH: A similar issue on SpecGL.
LR: This deals with the comment I raised about providing an overview of the 
test development process, addressing general concepts and the test 
development process.
PC: Are you referring to the question of what is conformance testing?
LH: This kind of information should not be part of this document. The 
intent of this document is not to be a tutorial.
MS: It should be part of the scope.
KD: Recommend modeling the introduction after the WCAG introduction.  A 
primer should make it easier to read the specification, not understand 
(explain) the specification.
DM: QA framework introduction could be used.
LR: Close the issue. The scope statement should include general 
clarification for the document.
Issue closed.
Issue 107: Differentiate between test materials for CR and those for Recs.
LR: Explained her concern on TestGl not addressing a test suite development 
based on the state of the specification. Writing a test suite for CR might 
have a different objective than writing a tests for a recommendation. For 
example: for CR the objective may be to test the specification, whereas a 
test suite for a Recommendation targets interoperability of the 
implementations. The TestGl is not allowing flexibility for CR.
DM: Verbiage should be included in the document to cover these areas.   The 
test suite development is a constant process of checking the specification, 
one thing that may occur, is that a test case identifies a problem in the 
specification. This could be an example if identifying problems in a CR.
SM:  The process of developing test cases for a test suite should not 
change; the working group is responsible for identifying the areas to test 
based on a particular state in the specification development process.
LR: I agree, but the way this document is written it does not allow for 
CR.  Something should be included in the introduction or scope. I am just 
not happy with the structure as it stands right now.
LH: A checkpoint will be appropriate to address this issue.
PC: It is a matter of coverage and completeness. Coverage is barely defined 
in the TestGl.
LR: It is up to the working group to determine the coverage. I am not 
particular on how is done, usually test suites for CR are smaller and more 
focused.
DM: WG delivers only one test suite. Sounds like you are talking about 
different test suites. Recommend adding a checkpoint.
LR: A WG may have multiple test suites for different parts of a 
specification or for different applications of the specification. The 
introduction or scope should also include something about the fact that a 
working group might have multiple test suites for a particular specification.
LH  Propose to accept Lynne’s proposal of adding a checkpoint to cover this 
issue.   The components of what needs to be address should included in the 
checkpoint
Issue Closed. Lynne will write a checkpoint to for resolution to this issue.
  Issue 23:  Should tests of MAY/SHOULD assertions be part of a conformance 
suite?
This issue was discussed at the f2f March meeting, after more discussion it 
was decided that Mark Skall will raise the issue in SpecGL to nail down the 
circumstances and David will to try to draft a resolution.
Issue closed.

Outreach report
David Marston presented the results of the XQuery and XSL outreach. There 
were two areas that generated discussion; reporting results and test 
assertions. It was recommended to have an example of a disclaimer or a 
boiler plate to capture the results. In regards to the test assertions a 
discussion on completeness was generated. Tagging was believed to raise 
concerns, some words are more important than others. David added that if 
this is an issue for some people it could be something we need to recheck. 
The concern was centered on the side of the test assertions that ties to 
the specification, no much was discussed in the area of the test assertions 
leading to test cases. Other than this, the group was basically asking 
clarifying questions.
PC: It is a complex process to identify assertions, it is a major task.
DM:  Complexity of expression and recursion can complicate things even more.
Lynne’s comments on TestGL.
Comment 1:  Prior to the Guideline section, provide an overview of the test 
development process, addressing general concepts and the deliverable path, 
i.e., spec test assertions test cases reporting maintenance. If possible 
the guidelines should follow this progression. Also, describe the key 
aspects of a test suite: traceability, verdict criteria, self-explanatory, 
valid, short/atomic.
This comment was discussed during the issues section (79) it was agreed 
that the scope statement should include general clarification.
Comments 2: The importance and need for traceability back to the spec must 
be clearly conveyed.
The comment was accepted by the group.
Comment 3: Make clear that there is no order to satisfying the CPs, as long 
as at the end of the day, they are satisfied.
This comment is making reference to the order of satisfying the checkpoint. 
Lynne added that it is when the test suite is finished, that all the 
checkpoints need to be satisfied. She further recommended that we should 
make that statement up front. The group agreed.

Comment 4: Although we can’t assume that the OpsGL and SpecGL were 
followed, but if they were, some of the CPs may already be satisfied we 
should provide a table cross-referencing where this occurs.
Lynne explained that we can not assume that a checkpoint was already done. 
The editors should include a note to make a table for cross-reference. 
Given the status of document a note should be included indicating that it 
was already done.
LH : This table is not trustworthy until LC.
Comment 5: Since we advocate developing test materials for CR, we should 
explain that test materials for CR may serve a different purpose and have 
different coverage than test materials for Rec and this is O.K.
         Already discussed as part of issue 107.
Comment 6: Can we incorporate some of the ideas from other WGs SVG and CSS 
have good test documentation, explaining not only how to build tests, but some
of the test suite principles.
         Lofton recommended there be a list of possible discrete 
deliverables in TTF. The group agreed.
Comment 7:  Although the formatting changes necessary for the GL and CPs 
will help, try to keep it simple.
  Lynne added that she had a difficult time with the order of the 
guideline. We should adopt the logical process from the tester. Organize it 
better.
PF: This is what GL 1 does, probably needs to be made more explicit. 
Changing the wording of the guideline will help.
Comment 8:  Suggest a separate Guideline for Test Results and 
Reporting.  It should also mention something about EARL.
LR: All this stuff got merged at the end.  It should stand out, there are 
also too many check points.
DM: What kind of action you think is expected from the results?
LH:  Providing a result management tool, then include an example in 
Ex/Tech. Also, the guideline should not mention EARL.
LR: There is more to say at a higher level related to the sorting and 
reporting capabilities in ck 5.2. A checkpoint should suggest reporting of 
result, something to make reporting consistent across implementations.
KD: More of describing what to do and how is organized.
LR: Also have a concern with the word framework in the title of guideline 
5, it seems to be focusing on recording a framework.
Comment 9: CP 1.1 Identify the target set of specifications being 
tested.  How extensive does this set need to be? For example, DOM must use 
valid components of other specifications, so would it be necessary to list 
all those specifications? Is this target set limited to those specifications
that are explicitly being tested rather than those specs that are utilized?

Everyone agreed that the language need to be worked out and the checkpoint 
need to be formatted the same way the SpecGL and OpsGL was done. The whole 
guideline needs to be restructured.
Comment 10: CP1.5, 1.6, 1.7, 1.8, 1.9: Related to discretionary choices. 
These CPs are related and breaking them into separate CPs has resulted in 
confusion as to the difference between them What is the difference between 
“defined ambiguously” (cp1.7) and “contradictory behaviors” {1.9)? Suggest 
combining, as: Identify behavior: undefined, ambiguous, and contradictory.

Patrick will circulate set suggested list of attributes that will lead to 
combining the checkpoints. Lofton expressed concern about the priorities, 
but David commented that after combining the checkpoint will become a 
priority one. Patrick added that the way this is going to be presented is 
not imposing is providing choices.
Comment 11: CP 2.1 Document the structure for the test suite and GL3 
Document the testing methodology. What is the difference between these? I 
think there is a difference, but my simple mind is having trouble making 
sense of all this.

  Patrick agreed that the distinction is not clear and suggested combining 
GL-2 and 3 and speaking more in term of the language.  If not combined they 
should be reversed in order.
Peter took the action to combine the Guidelines.
Comment 12: CP 3.2 Identify publicly available testing techniques. What 
does ‘testing techniques’ mean? How is this different from test automation 
and framework?  How much of a search for these techniques needs to be done?
Comment is moot due to previous action.
Comment 13: CP 4.1 List available test frameworks and applicable automation 
and justify why new frameworks are needed…..
a) Define these terms and their scope. How is this different from 3.2?
b) Why must a justification be given who is the justification for? This may 
be adding extra work on the WG with minimal if any benefit.

Lynne added that the default is not to provide automated frameworks and 
that we are being unrealistic. Patrick suggested that all the automation 
checkpoints need to be collapsed into one checkpoint; he added that if we 
are going to do automation we need to provide good guidelines. Lynne also 
suggested using the ideas from the VML presentation to include test case 
management.
The group agreed to restructure GL 4 to include automation and test case 
management.
Comment 13- 17
These comments are going to be covered by the restructure of the guideline 
4, mapping the process from beginning to end by looking at the structure. 
Patrick took this action. Action due two weeks from Monday.
Comment 18: CP 5.2 Ensure the ease of use for results reporting. 
Demonstrate results reporting has sorting and filtering. Why is this P1? 
Although nice to have, why is sorting and filtering required?

Comment already discussed.



At 07:25 AM 3/10/2003 -0700, you wrote:
>Hi,
>
>Thanks for sending this, Sandra.
>
>I noticed that it did not get relayed by the www-qa-wg@w3.org list 
>manager, and it is not in the archive.  I suspect that the reason is that 
>your name looks a little different than normal (sandra.martinez@nist.gov), 
>and the list manager didn't recognize you.
>
>Could you please resend?  Also, if you can convert to HTML, that would be 
>better.  Olivier can't or doesn't want to handle .DOC.  I think Lynne has 
>a resource for doing this (which is why I copied her).
>
>Thanks,
>-Lofton.
>
>
>
>>X-Authentication-Warning: calendar.nist.gov: nobody set sender to 
>>martines@nist.gov using -f
>>Date: Sun,  9 Mar 2003 20:39:52 -0500
>>From: martines@nist.gov
>>To: Lofton Henderson <lofton@rockynet.com>
>>Cc: www-qa-wg@w3.org
>>Subject: Re: raw draft minutes
>>User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
>>X-RCPT-TO: <lofton@rockynet.com>
>>
>>Included is the draft minutes for Thursday morning. Noticed that there 
>>are two
>>due dates missing in the action list.
>>
>>
>>Sandra
>>
>>Quoting Lofton Henderson <lofton@rockynet.com>:
>>
>> >
>> > It would be useful (to me, at least) to have access to raw draft minutes
>> > from Boston (4 half days).  Could you minute takers please send them 
>> to the
>> >
>> > WG list?
>> >
>> > Thanks,
>> > -Lofton.
>> >
>> >
>> >
>>
>
>

Sandra I. Martinez
National Institute of Standards and Technology
100 Bureau Drive, Stop 8970,
Gaithersburg, Md. 20899

(301) 975-3579
sandra.martinez@nist.gov

Received on Monday, 10 March 2003 10:32:32 UTC