Re: Fwd: Re: raw draft minutes

[Attn David and Patrick...]

At 10:22 AM 3/10/03 -0500, Sandra Martinez wrote:

>Included are the draft minutes for Thursday morning. Noticed that there 
>are two
>due dates missing in the action list.
>
>[...]
>Summary of New Action Items:
>
>A-2003-03-07-1: Action to Peter to send comments to the  form for OpsGL. 
>By Tuesday, March 10.
>A-2003-03-07-2: Action to Lynne to include Ian Jacob’s comments into the 
>form for SpecGL. By Tuesday, March 10.
>A-2002-03-07-3: Action to Patrick to review to QAWG charter. By end of 
>next week
>A-2002-03-07-4: Action to Lofton Updating the QA PD template and message 
>to chair list  March 20
>A-2002-03-07-5: Action to David to draft resolution to issue 23.

David, would you propose a date for this please?

>A-2002-03-07-6: Action to Lynne to write checkpoint to reflect resolution 
>of   issue 107 march 20
>A-2002-03-07-7: Action to Patrick to circulate a set of suggested list of 
>attributes that will lead to collapsing the checkpoints in GL 1.

Patrick, would you propose a date for this please? (Is it done already?)

>A-2003-03-07-8: Action to Peter to combine GL 2 and 3. By Friday, March 14.
>A-2003-03-07-9: Action to Patrick to restructure of the guideline 4.

For Olivier's benefit, below shows that the date is "two weeks from Monday" 
(April 24).

All for now,
-Lofton.




>Peter initiated the meeting providing a brief summary of the results from 
>the break out team’s review of the Process document using the process 
>document template.  The activity resulted in a set of comments that are 
>going to be incorporate in the OpsGL comments form. Patrick took this action.
>Lynne will incorporate Ian Jacobs’s comments into the SpecGL comments form.
>
>As a result of this review, Lofton recommended that a review of the 
>charter should also be done. Patrick took the action of performing the review.
>
>TestGL issues resolution
>
>Issue 13: Is inter-standard or multi-standard interoperability within the 
>QA scope? Issue raised in email thread (DM, 2001-11-07, 2nd pgph).
>         MS: There are no requirements for interoperability. We need 
> specific checkpoints.
>LR: The requirements for interoperability are being addressed in other 
>specifications. If they are represented in those specifications they are 
>going to be tested.
>
>         LF: This is an old issue, just noticed the e-mail that was generated.
>
>         DM: The intention of this issue was related to the pipeline.
>
>         MS: There is a problem with the way we are capturing the issues.
>LR: This type of information should not be included in the TestGl at this 
>point. We need to keep this document simple.
>
>KD  Agree. The TestGl needs to be simple and this will not prevent writing 
>a separate document to address clarification items.
>
>PC  Also agree to keep the document simple. Put some words in the document 
>about extensibility/optionality. Propose to address this in the SpecGl, 
>not in TestGl.
>
>LH : Propose to cut and paste clarification into the body of the 
>issue.  Also propose to go with Lynne’s proposal.
>The issue is closed.
>
>Issue 79: Should Test Guidelines address how to write a good test suite?
>
>         LH: A similar issue on SpecGL.
>LR: This deals with the comment I raised about providing an overview of 
>the test development process, addressing general concepts and the test 
>development process.
>PC: Are you referring to the question of what is conformance testing?
>LH: This kind of information should not be part of this document. The 
>intent of this document is not to be a tutorial.
>MS: It should be part of the scope.
>KD: Recommend modeling the introduction after the WCAG introduction.  A 
>primer should make it easier to read the specification, not understand 
>(explain) the specification.
>DM: QA framework introduction could be used.
>LR: Close the issue. The scope statement should include general 
>clarification for the document.
>Issue closed.
>Issue 107: Differentiate between test materials for CR and those for Recs.
>LR: Explained her concern on TestGl not addressing a test suite 
>development based on the state of the specification. Writing a test suite 
>for CR might have a different objective than writing a tests for a 
>recommendation. For example: for CR the objective may be to test the 
>specification, whereas a test suite for a Recommendation targets 
>interoperability of the implementations. The TestGl is not allowing 
>flexibility for CR.
>DM: Verbiage should be included in the document to cover these 
>areas.   The test suite development is a constant process of checking the 
>specification, one thing that may occur, is that a test case identifies a 
>problem in the specification. This could be an example if identifying 
>problems in a CR.
>SM:  The process of developing test cases for a test suite should not 
>change; the working group is responsible for identifying the areas to test 
>based on a particular state in the specification development process.
>LR: I agree, but the way this document is written it does not allow for 
>CR.  Something should be included in the introduction or scope. I am just 
>not happy with the structure as it stands right now.
>LH: A checkpoint will be appropriate to address this issue.
>PC: It is a matter of coverage and completeness. Coverage is barely 
>defined in the TestGl.
>LR: It is up to the working group to determine the coverage. I am not 
>particular on how is done, usually test suites for CR are smaller and more 
>focused.
>DM: WG delivers only one test suite. Sounds like you are talking about 
>different test suites. Recommend adding a checkpoint.
>LR: A WG may have multiple test suites for different parts of a 
>specification or for different applications of the specification. The 
>introduction or scope should also include something about the fact that a 
>working group might have multiple test suites for a particular specification.
>LH  Propose to accept Lynne’s proposal of adding a checkpoint to cover 
>this issue.   The components of what needs to be address should included 
>in the checkpoint
>Issue Closed. Lynne will write a checkpoint to for resolution to this issue.
>  Issue 23:  Should tests of MAY/SHOULD assertions be part of a 
> conformance suite?
>This issue was discussed at the f2f March meeting, after more discussion 
>it was decided that Mark Skall will raise the issue in SpecGL to nail down 
>the circumstances and David will to try to draft a resolution.
>Issue closed.
>
>Outreach report
>David Marston presented the results of the XQuery and XSL outreach. There 
>were two areas that generated discussion; reporting results and test 
>assertions. It was recommended to have an example of a disclaimer or a 
>boiler plate to capture the results. In regards to the test assertions a 
>discussion on completeness was generated. Tagging was believed to raise 
>concerns, some words are more important than others. David added that if 
>this is an issue for some people it could be something we need to recheck. 
>The concern was centered on the side of the test assertions that ties to 
>the specification, no much was discussed in the area of the test 
>assertions leading to test cases. Other than this, the group was basically 
>asking clarifying questions.
>PC: It is a complex process to identify assertions, it is a major task.
>DM:  Complexity of expression and recursion can complicate things even more.
>Lynne’s comments on TestGL.
>Comment 1:  Prior to the Guideline section, provide an overview of the 
>test development process, addressing general concepts and the deliverable 
>path, i.e., spec test assertions test cases reporting maintenance. If 
>possible the guidelines should follow this progression. Also, describe the 
>key aspects of a test suite: traceability, verdict criteria, 
>self-explanatory, valid, short/atomic.
>This comment was discussed during the issues section (79) it was agreed 
>that the scope statement should include general clarification.
>Comments 2: The importance and need for traceability back to the spec must 
>be clearly conveyed.
>The comment was accepted by the group.
>Comment 3: Make clear that there is no order to satisfying the CPs, as 
>long as at the end of the day, they are satisfied.
>This comment is making reference to the order of satisfying the 
>checkpoint. Lynne added that it is when the test suite is finished, that 
>all the checkpoints need to be satisfied. She further recommended that we 
>should make that statement up front. The group agreed.
>
>Comment 4: Although we can’t assume that the OpsGL and SpecGL were 
>followed, but if they were, some of the CPs may already be satisfied we 
>should provide a table cross-referencing where this occurs.
>Lynne explained that we can not assume that a checkpoint was already done. 
>The editors should include a note to make a table for cross-reference. 
>Given the status of document a note should be included indicating that it 
>was already done.
>LH : This table is not trustworthy until LC.
>Comment 5: Since we advocate developing test materials for CR, we should 
>explain that test materials for CR may serve a different purpose and have 
>different coverage than test materials for Rec and this is O.K.
>         Already discussed as part of issue 107.
>Comment 6: Can we incorporate some of the ideas from other WGs SVG and CSS 
>have good test documentation, explaining not only how to build tests, but some
>of the test suite principles.
>         Lofton recommended there be a list of possible discrete 
> deliverables in TTF. The group agreed.
>Comment 7:  Although the formatting changes necessary for the GL and CPs 
>will help, try to keep it simple.
>  Lynne added that she had a difficult time with the order of the 
> guideline. We should adopt the logical process from the tester. Organize 
> it better.
>PF: This is what GL 1 does, probably needs to be made more explicit. 
>Changing the wording of the guideline will help.
>Comment 8:  Suggest a separate Guideline for Test Results and 
>Reporting.  It should also mention something about EARL.
>LR: All this stuff got merged at the end.  It should stand out, there are 
>also too many check points.
>DM: What kind of action you think is expected from the results?
>LH:  Providing a result management tool, then include an example in 
>Ex/Tech. Also, the guideline should not mention EARL.
>LR: There is more to say at a higher level related to the sorting and 
>reporting capabilities in ck 5.2. A checkpoint should suggest reporting of 
>result, something to make reporting consistent across implementations.
>KD: More of describing what to do and how is organized.
>LR: Also have a concern with the word framework in the title of guideline 
>5, it seems to be focusing on recording a framework.
>Comment 9: CP 1.1 Identify the target set of specifications being 
>tested.  How extensive does this set need to be? For example, DOM must use 
>valid components of other specifications, so would it be necessary to list 
>all those specifications? Is this target set limited to those specifications
>that are explicitly being tested rather than those specs that are utilized?
>
>Everyone agreed that the language need to be worked out and the checkpoint 
>need to be formatted the same way the SpecGL and OpsGL was done. The whole 
>guideline needs to be restructured.
>Comment 10: CP1.5, 1.6, 1.7, 1.8, 1.9: Related to discretionary choices. 
>These CPs are related and breaking them into separate CPs has resulted in 
>confusion as to the difference between them What is the difference between 
>“defined ambiguously” (cp1.7) and “contradictory behaviors” {1.9)? Suggest 
>combining, as: Identify behavior: undefined, ambiguous, and contradictory.
>
>Patrick will circulate set suggested list of attributes that will lead to 
>combining the checkpoints. Lofton expressed concern about the priorities, 
>but David commented that after combining the checkpoint will become a 
>priority one. Patrick added that the way this is going to be presented is 
>not imposing is providing choices.
>Comment 11: CP 2.1 Document the structure for the test suite and GL3 
>Document the testing methodology. What is the difference between these? I 
>think there is a difference, but my simple mind is having trouble making 
>sense of all this.
>
>  Patrick agreed that the distinction is not clear and suggested combining 
> GL-2 and 3 and speaking more in term of the language.  If not combined 
> they should be reversed in order.
>Peter took the action to combine the Guidelines.
>Comment 12: CP 3.2 Identify publicly available testing techniques. What 
>does ‘testing techniques’ mean? How is this different from test automation 
>and framework?  How much of a search for these techniques needs to be done?
>Comment is moot due to previous action.
>Comment 13: CP 4.1 List available test frameworks and applicable 
>automation and justify why new frameworks are needed…..
>a) Define these terms and their scope. How is this different from 3.2?
>b) Why must a justification be given who is the justification for? This 
>may be adding extra work on the WG with minimal if any benefit.
>
>Lynne added that the default is not to provide automated frameworks and 
>that we are being unrealistic. Patrick suggested that all the automation 
>checkpoints need to be collapsed into one checkpoint; he added that if we 
>are going to do automation we need to provide good guidelines. Lynne also 
>suggested using the ideas from the VML presentation to include test case 
>management.
>The group agreed to restructure GL 4 to include automation and test case 
>management.
>Comment 13- 17
>These comments are going to be covered by the restructure of the guideline 
>4, mapping the process from beginning to end by looking at the structure. 
>Patrick took this action. Action due two weeks from Monday.
>Comment 18: CP 5.2 Ensure the ease of use for results reporting. 
>Demonstrate results reporting has sorting and filtering. Why is this P1? 
>Although nice to have, why is sorting and filtering required?
>
>Comment already discussed.
>
>
>
>At 07:25 AM 3/10/2003 -0700, you wrote:
>>Hi,
>>
>>Thanks for sending this, Sandra.
>>
>>I noticed that it did not get relayed by the www-qa-wg@w3.org list 
>>manager, and it is not in the archive.  I suspect that the reason is that 
>>your name looks a little different than normal 
>>(sandra.martinez@nist.gov), and the list manager didn't recognize you.
>>
>>Could you please resend?  Also, if you can convert to HTML, that would be 
>>better.  Olivier can't or doesn't want to handle .DOC.  I think Lynne has 
>>a resource for doing this (which is why I copied her).
>>
>>Thanks,
>>-Lofton.
>>
>>
>>
>>>X-Authentication-Warning: calendar.nist.gov: nobody set sender to 
>>>martines@nist.gov using -f
>>>Date: Sun,  9 Mar 2003 20:39:52 -0500
>>>From: martines@nist.gov
>>>To: Lofton Henderson <lofton@rockynet.com>
>>>Cc: www-qa-wg@w3.org
>>>Subject: Re: raw draft minutes
>>>User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
>>>X-RCPT-TO: <lofton@rockynet.com>
>>>
>>>Included is the draft minutes for Thursday morning. Noticed that there 
>>>are two
>>>due dates missing in the action list.
>>>
>>>
>>>Sandra
>>>
>>>Quoting Lofton Henderson <lofton@rockynet.com>:
>>>
>>> >
>>> > It would be useful (to me, at least) to have access to raw draft minutes
>>> > from Boston (4 half days).  Could you minute takers please send them 
>>> to the
>>> >
>>> > WG list?
>>> >
>>> > Thanks,
>>> > -Lofton.
>>> >
>>> >
>>> >
>>
>
>Sandra I. Martinez
>National Institute of Standards and Technology
>100 Bureau Drive, Stop 8970,
>Gaithersburg, Md. 20899
>
>(301) 975-3579
>sandra.martinez@nist.gov
>
>

Received on Monday, 10 March 2003 15:17:04 UTC