Re: Fwd: Re: raw draft minutes

I will circulate my list of assertion attributes by this Friday (March 14).

Lofton Henderson wrote:

>
> [Attn David and Patrick...]
>
> At 10:22 AM 3/10/03 -0500, Sandra Martinez wrote:
>
>> Included are the draft minutes for Thursday morning. Noticed that 
>> there are two
>> due dates missing in the action list.
>>
>> [...]
>> Summary of New Action Items:
>>
>> A-2003-03-07-1: Action to Peter to send comments to the form for 
>> OpsGL. By Tuesday, March 10.
>> A-2003-03-07-2: Action to Lynne to include Ian Jacobs comments into 
>> the form for SpecGL. By Tuesday, March 10.
>> A-2002-03-07-3: Action to Patrick to review to QAWG charter. By end 
>> of next week
>> A-2002-03-07-4: Action to Lofton Updating the QA PD template and 
>> message to chair list March 20
>> A-2002-03-07-5: Action to David to draft resolution to issue 23.
>
>
> David, would you propose a date for this please?
>
>> A-2002-03-07-6: Action to Lynne to write checkpoint to reflect 
>> resolution of issue 107 march 20
>> A-2002-03-07-7: Action to Patrick to circulate a set of suggested 
>> list of attributes that will lead to collapsing the checkpoints in GL 1.
>
>
> Patrick, would you propose a date for this please? (Is it done already?)
>
>> A-2003-03-07-8: Action to Peter to combine GL 2 and 3. By Friday, 
>> March 14.
>> A-2003-03-07-9: Action to Patrick to restructure of the guideline 4.
>
>
> For Olivier's benefit, below shows that the date is "two weeks from 
> Monday" (April 24).
>
> All for now,
> -Lofton.
>
>
>
>
>> Peter initiated the meeting providing a brief summary of the results 
>> from the break out teams review of the Process document using the 
>> process document template. The activity resulted in a set of comments 
>> that are going to be incorporate in the OpsGL comments form. Patrick 
>> took this action.
>> Lynne will incorporate Ian Jacobss comments into the SpecGL comments 
>> form.
>>
>> As a result of this review, Lofton recommended that a review of the 
>> charter should also be done. Patrick took the action of performing 
>> the review.
>>
>> TestGL issues resolution
>>
>> Issue 13: Is inter-standard or multi-standard interoperability within 
>> the QA scope? Issue raised in email thread (DM, 2001-11-07, 2nd pgph).
>> MS: There are no requirements for interoperability. We need specific 
>> checkpoints.
>> LR: The requirements for interoperability are being addressed in 
>> other specifications. If they are represented in those specifications 
>> they are going to be tested.
>>
>> LF: This is an old issue, just noticed the e-mail that was generated.
>>
>> DM: The intention of this issue was related to the pipeline.
>>
>> MS: There is a problem with the way we are capturing the issues.
>> LR: This type of information should not be included in the TestGl at 
>> this point. We need to keep this document simple.
>>
>> KD Agree. The TestGl needs to be simple and this will not prevent 
>> writing a separate document to address clarification items.
>>
>> PC Also agree to keep the document simple. Put some words in the 
>> document about extensibility/optionality. Propose to address this in 
>> the SpecGl, not in TestGl.
>>
>> LH : Propose to cut and paste clarification into the body of the 
>> issue. Also propose to go with Lynnes proposal.
>> The issue is closed.
>>
>> Issue 79: Should Test Guidelines address how to write a good test suite?
>>
>> LH: A similar issue on SpecGL.
>> LR: This deals with the comment I raised about providing an overview 
>> of the test development process, addressing general concepts and the 
>> test development process.
>> PC: Are you referring to the question of what is conformance testing?
>> LH: This kind of information should not be part of this document. The 
>> intent of this document is not to be a tutorial.
>> MS: It should be part of the scope.
>> KD: Recommend modeling the introduction after the WCAG introduction. 
>> A primer should make it easier to read the specification, not 
>> understand (explain) the specification.
>> DM: QA framework introduction could be used.
>> LR: Close the issue. The scope statement should include general 
>> clarification for the document.
>> Issue closed.
>> Issue 107: Differentiate between test materials for CR and those for 
>> Recs.
>> LR: Explained her concern on TestGl not addressing a test suite 
>> development based on the state of the specification. Writing a test 
>> suite for CR might have a different objective than writing a tests 
>> for a recommendation. For example: for CR the objective may be to 
>> test the specification, whereas a test suite for a Recommendation 
>> targets interoperability of the implementations. The TestGl is not 
>> allowing flexibility for CR.
>> DM: Verbiage should be included in the document to cover these areas. 
>> The test suite development is a constant process of checking the 
>> specification, one thing that may occur, is that a test case 
>> identifies a problem in the specification. This could be an example 
>> if identifying problems in a CR.
>> SM: The process of developing test cases for a test suite should not 
>> change; the working group is responsible for identifying the areas to 
>> test based on a particular state in the specification development 
>> process.
>> LR: I agree, but the way this document is written it does not allow 
>> for CR. Something should be included in the introduction or scope. I 
>> am just not happy with the structure as it stands right now.
>> LH: A checkpoint will be appropriate to address this issue.
>> PC: It is a matter of coverage and completeness. Coverage is barely 
>> defined in the TestGl.
>> LR: It is up to the working group to determine the coverage. I am not 
>> particular on how is done, usually test suites for CR are smaller and 
>> more focused.
>> DM: WG delivers only one test suite. Sounds like you are talking 
>> about different test suites. Recommend adding a checkpoint.
>> LR: A WG may have multiple test suites for different parts of a 
>> specification or for different applications of the specification. The 
>> introduction or scope should also include something about the fact 
>> that a working group might have multiple test suites for a particular 
>> specification.
>> LH Propose to accept Lynnes proposal of adding a checkpoint to cover 
>> this issue. The components of what needs to be address should 
>> included in the checkpoint
>> Issue Closed. Lynne will write a checkpoint to for resolution to this 
>> issue.
>> Issue 23: Should tests of MAY/SHOULD assertions be part of a 
>> conformance suite?
>> This issue was discussed at the f2f March meeting, after more 
>> discussion it was decided that Mark Skall will raise the issue in 
>> SpecGL to nail down the circumstances and David will to try to draft 
>> a resolution.
>> Issue closed.
>>
>> Outreach report
>> David Marston presented the results of the XQuery and XSL outreach. 
>> There were two areas that generated discussion; reporting results and 
>> test assertions. It was recommended to have an example of a 
>> disclaimer or a boiler plate to capture the results. In regards to 
>> the test assertions a discussion on completeness was generated. 
>> Tagging was believed to raise concerns, some words are more important 
>> than others. David added that if this is an issue for some people it 
>> could be something we need to recheck. The concern was centered on 
>> the side of the test assertions that ties to the specification, no 
>> much was discussed in the area of the test assertions leading to test 
>> cases. Other than this, the group was basically asking clarifying 
>> questions.
>> PC: It is a complex process to identify assertions, it is a major task.
>> DM: Complexity of expression and recursion can complicate things even 
>> more.
>> Lynnes comments on TestGL.
>> Comment 1: Prior to the Guideline section, provide an overview of the 
>> test development process, addressing general concepts and the 
>> deliverable path, i.e., spec test assertions test cases reporting 
>> maintenance. If possible the guidelines should follow this 
>> progression. Also, describe the key aspects of a test suite: 
>> traceability, verdict criteria, self-explanatory, valid, short/atomic.
>> This comment was discussed during the issues section (79) it was 
>> agreed that the scope statement should include general clarification.
>> Comments 2: The importance and need for traceability back to the spec 
>> must be clearly conveyed.
>> The comment was accepted by the group.
>> Comment 3: Make clear that there is no order to satisfying the CPs, 
>> as long as at the end of the day, they are satisfied.
>> This comment is making reference to the order of satisfying the 
>> checkpoint. Lynne added that it is when the test suite is finished, 
>> that all the checkpoints need to be satisfied. She further 
>> recommended that we should make that statement up front. The group 
>> agreed.
>>
>> Comment 4: Although we cant assume that the OpsGL and SpecGL were 
>> followed, but if they were, some of the CPs may already be satisfied 
>> we should provide a table cross-referencing where this occurs.
>> Lynne explained that we can not assume that a checkpoint was already 
>> done. The editors should include a note to make a table for 
>> cross-reference. Given the status of document a note should be 
>> included indicating that it was already done.
>> LH : This table is not trustworthy until LC.
>> Comment 5: Since we advocate developing test materials for CR, we 
>> should explain that test materials for CR may serve a different 
>> purpose and have different coverage than test materials for Rec and 
>> this is O.K.
>> Already discussed as part of issue 107.
>> Comment 6: Can we incorporate some of the ideas from other WGs SVG 
>> and CSS have good test documentation, explaining not only how to 
>> build tests, but some
>> of the test suite principles.
>> Lofton recommended there be a list of possible discrete deliverables 
>> in TTF. The group agreed.
>> Comment 7: Although the formatting changes necessary for the GL and 
>> CPs will help, try to keep it simple.
>> Lynne added that she had a difficult time with the order of the 
>> guideline. We should adopt the logical process from the tester. 
>> Organize it better.
>> PF: This is what GL 1 does, probably needs to be made more explicit. 
>> Changing the wording of the guideline will help.
>> Comment 8: Suggest a separate Guideline for Test Results and 
>> Reporting. It should also mention something about EARL.
>> LR: All this stuff got merged at the end. It should stand out, there 
>> are also too many check points.
>> DM: What kind of action you think is expected from the results?
>> LH: Providing a result management tool, then include an example in 
>> Ex/Tech. Also, the guideline should not mention EARL.
>> LR: There is more to say at a higher level related to the sorting and 
>> reporting capabilities in ck 5.2. A checkpoint should suggest 
>> reporting of result, something to make reporting consistent across 
>> implementations.
>> KD: More of describing what to do and how is organized.
>> LR: Also have a concern with the word framework in the title of 
>> guideline 5, it seems to be focusing on recording a framework.
>> Comment 9: CP 1.1 Identify the target set of specifications being 
>> tested. How extensive does this set need to be? For example, DOM must 
>> use valid components of other specifications, so would it be 
>> necessary to list all those specifications? Is this target set 
>> limited to those specifications
>> that are explicitly being tested rather than those specs that are 
>> utilized?
>>
>> Everyone agreed that the language need to be worked out and the 
>> checkpoint need to be formatted the same way the SpecGL and OpsGL was 
>> done. The whole guideline needs to be restructured.
>> Comment 10: CP1.5, 1.6, 1.7, 1.8, 1.9: Related to discretionary 
>> choices. These CPs are related and breaking them into separate CPs 
>> has resulted in confusion as to the difference between them What is 
>> the difference between defined ambiguously (cp1.7) and 
>> contradictory behaviors {1.9)? Suggest combining, as: Identify 
>> behavior: undefined, ambiguous, and contradictory.
>>
>> Patrick will circulate set suggested list of attributes that will 
>> lead to combining the checkpoints. Lofton expressed concern about the 
>> priorities, but David commented that after combining the checkpoint 
>> will become a priority one. Patrick added that the way this is going 
>> to be presented is not imposing is providing choices.
>> Comment 11: CP 2.1 Document the structure for the test suite and GL3 
>> Document the testing methodology. What is the difference between 
>> these? I think there is a difference, but my simple mind is having 
>> trouble making sense of all this.
>>
>> Patrick agreed that the distinction is not clear and suggested 
>> combining GL-2 and 3 and speaking more in term of the language. If 
>> not combined they should be reversed in order.
>> Peter took the action to combine the Guidelines.
>> Comment 12: CP 3.2 Identify publicly available testing techniques. 
>> What does testing techniques mean? How is this different from test 
>> automation and framework? How much of a search for these techniques 
>> needs to be done?
>> Comment is moot due to previous action.
>> Comment 13: CP 4.1 List available test frameworks and applicable 
>> automation and justify why new frameworks are needed&..
>> a) Define these terms and their scope. How is this different from 3.2?
>> b) Why must a justification be given who is the justification for? 
>> This may be adding extra work on the WG with minimal if any benefit.
>>
>> Lynne added that the default is not to provide automated frameworks 
>> and that we are being unrealistic. Patrick suggested that all the 
>> automation checkpoints need to be collapsed into one checkpoint; he 
>> added that if we are going to do automation we need to provide good 
>> guidelines. Lynne also suggested using the ideas from the VML 
>> presentation to include test case management.
>> The group agreed to restructure GL 4 to include automation and test 
>> case management.
>> Comment 13- 17
>> These comments are going to be covered by the restructure of the 
>> guideline 4, mapping the process from beginning to end by looking at 
>> the structure. Patrick took this action. Action due two weeks from 
>> Monday.
>> Comment 18: CP 5.2 Ensure the ease of use for results reporting. 
>> Demonstrate results reporting has sorting and filtering. Why is this 
>> P1? Although nice to have, why is sorting and filtering required?
>>
>> Comment already discussed.
>>
>>
>>
>> At 07:25 AM 3/10/2003 -0700, you wrote:
>>
>>> Hi,
>>>
>>> Thanks for sending this, Sandra.
>>>
>>> I noticed that it did not get relayed by the www-qa-wg@w3.org list 
>>> manager, and it is not in the archive. I suspect that the reason is 
>>> that your name looks a little different than normal 
>>> (sandra.martinez@nist.gov), and the list manager didn't recognize you.
>>>
>>> Could you please resend? Also, if you can convert to HTML, that 
>>> would be better. Olivier can't or doesn't want to handle .DOC. I 
>>> think Lynne has a resource for doing this (which is why I copied her).
>>>
>>> Thanks,
>>> -Lofton.
>>>
>>>
>>>
>>>> X-Authentication-Warning: calendar.nist.gov: nobody set sender to 
>>>> martines@nist.gov using -f
>>>> Date: Sun, 9 Mar 2003 20:39:52 -0500
>>>> From: martines@nist.gov
>>>> To: Lofton Henderson <lofton@rockynet.com>
>>>> Cc: www-qa-wg@w3.org
>>>> Subject: Re: raw draft minutes
>>>> User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
>>>> X-RCPT-TO: <lofton@rockynet.com>
>>>>
>>>> Included is the draft minutes for Thursday morning. Noticed that 
>>>> there are two
>>>> due dates missing in the action list.
>>>>
>>>>
>>>> Sandra
>>>>
>>>> Quoting Lofton Henderson <lofton@rockynet.com>:
>>>>
>>>> >
>>>> > It would be useful (to me, at least) to have access to raw draft 
>>>> minutes
>>>> > from Boston (4 half days). Could you minute takers please send 
>>>> them to the
>>>> >
>>>> > WG list?
>>>> >
>>>> > Thanks,
>>>> > -Lofton.
>>>> >
>>>> >
>>>> >
>>>
>>>
>>
>> Sandra I. Martinez
>> National Institute of Standards and Technology
>> 100 Bureau Drive, Stop 8970,
>> Gaithersburg, Md. 20899
>>
>> (301) 975-3579
>> sandra.martinez@nist.gov
>>
>>
>
>

Received on Monday, 10 March 2003 15:21:59 UTC