- From: Karl Dubost <karl@w3.org>
- Date: Thu, 17 Oct 2002 15:25:58 +0900
- To: www-qa-wg@w3.org
No comments so the Minutes are final QA Working Group Teleconference Wednesday, 25-09-2002 -- Scribe: Karl Dubost Attendees: (dd) Dimitris Dimitriadis (Ontologicon) (KD) Karl Dubost (W3C, WG co-chair) (DH) Dominique Hazael-Massieux (W3C) (LH) Lofton Henderson (CGMO - WG co-chair) (DM) David Marston (IBM) (KG) Kirill Gavrylyuk (Microsoft) Regrets: (LR) Lynne Rosenthal (NIST - IG co-chair) (MS) Mark Skall (NIST) (SM) Sandra Martinez (NIST) Absent: (PF) Peter Fawcett (RealNetworks) (AT) Andrew Thackrah (Open Group) (JM) Jack Morrison (Sun) Summary of New Action Items: None Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2002Sep/0065.html Previous Telcon Minutes: http://lists.w3.org/Archives/Public/www-qa-wg/2002Sep/0068.html Lofton: Spec GL (follow-up on previous meeting, overdue agenda item). Karl: This is an issue with Spec GL which is very crucial part of the QA Framework. The Spec GL are not clear enough with an external point of view, it's really hard to if you don't know all the discussions we have been having, it's really difficult to read. Who will apply the Spec GL on themselves? Dom: Mark is supposed to do so Karl: It's really important to have a new look at them. One idea would be to move the examples from the SpeC GL to the ExTechs. A lighter spec easy to use, fast to read. Let's keep in mind that people reading the Spec GL won't be aware of the whole discussion behind that. It's really boring to read them right now, having a better logic behind it would help to understand and make it more usable for editors. It seems hard to understand and too constraintful right now and enough inviting. Lofton: That looks like some of the issues that Alex pointed in his emails. Lynne seems to have the same feelings as well. She said she had a hard time reading it after 3 months with no contact with QA WG. That's a very important point of view because she is part of the WG. So I think we need to have a close look at clarity and simplicity. 2 more comments: Lynne is coeditor with Dominique and her role is to identify specific issues in the recent mails. I hence have 21 new issues to integrate in the issues list. Most of them came out of the mail exchange with Alex and some of them matches what you're talking about. Karl: I've asked someone in the Team who has never read a W3C spec to read the spec. The one who'll do that is an intern in W3C Team I was about to pull out the examples from the Spec GL and put them in the ExTechs doc. Lofton: That's what I was about to suggest and then, we can look at Spec GL, only keeping some of them used for illustrations to remove ambiguity and that will open up a lot of issues esp. whether specs should have examples since it should apply to us as much as other specs. Dom: I think we actually need to change this checkpoint as suggested Alex, maybe somehitng "give an example for each ambiguous req". Lofton: we'll need to think more on this, but later next week, we'll have a new block of identified issues that'll help getting ready for tokyo. Some are editorial, some others are at the heart of the DOV model. Karl, your point is well taken. You should proceed and put the examples from Spec GL into extechs. I personnally agree with lots of comments you made. I think Alex made a lot of good points too. Dimitris: We should probably contact one or 2 editors of very heavy specs and ask them explicitely to comment on it. Lofton: should we ask on the current state? or wait for the next publication in one month? Dimitris: let's do that after next publication but let's do this! I'll try to approach chairs/editors unofficially to have at least some comments before Tokyo === Test GL draft and issues === Kirill presents the Test GL. http://www.w3.org/QA/WG/2002/framework-20020826/qaframe-test http://www.w3.org/QA/WG/qawg-issues-html.html#x79 There are not so many issues right now. The most common one has been raised by Alex. "How to create a good test suite". For each checkpoints, we say what's the best way to achieve this checkpoint. Lofton: In SpecGL, because of the imperative wording in many of the guidelines, it could seem that we ask for a process, more than the final result itself, which leads to unclarity. For this reason, we are sometimes misunderstood in the guidelines. In SpecGL, we do focus more on the results than the process, in the descriptions of the guidelines and checkpoints missed KG. Lofton: I don't necessary agree with Alex's TestGL point (issue #79). But I would try to clarify the spec to make it clear whether we are addressing process versus result in each particular guideline and checkpoint. Then we can argue the issue about whether it should be result only, or process also. Kirill: I understand. Ops GL and Test GL are very entighten together. Test GL are just a continuation of the process (Op GL). Lofton: if we look at the checkpoints and enumerate what things must be part of the TS, then we could rephrase it for example to have only the result in the checkpoint and move the process into the ExTech Kirill: gives an example for CK 1.1 Dimitris: We could rewrite checkpoints according to the distinction between how things need to be done and what must be in them , however editors should only investigate time if the distinction definitely holds Lofton: I think there's a distinction... Can we clarify for ourselves the nature of process versus results... and if we have done that, we will be able to solve the issues. One way of looking at it is to look at "what it should contain" as TS deliverables Dimitris: We must first decide what's the list of deliverables and with this list we will be able to write in a way that will be clear. Kirill: no objections. I would like to just go and propose reformulation of the guidelines. So we can let the old text and new text... and choose. Gd 4. How should we coordinate Ops and Test guidelines. Is it ok to duplicate checkpoints? Kirill: we do a lot of process in the Guideline 4 right now. Maybe after reformulation, we will not have anymore a duplication of the TEst GL. Issue 2 resolved after no objections. Ck 1.7 Identify optional behaviors in the specification. Do we keep this checkpoint or drop it? Kirill: We still haven't decide if we should skip this CK, or if it should go in the Spec Guidelines. Lofton: The test Suite has to know about optional behaviors. Do we mean discretionary behaviors. Kirill: No for examples you can have one binding and only one implementation for this particular binding. Lofton: Optional behaviors have to be identified in the TS or TS harness itself. I.e., the TS/harness must be aware of whether a test is for an optional behavior or not, and must be aware of whether the implementation claims to do it or not. There's something similar in the XSLT test suite. Kirill: is it a requirements or up to the implementations of the TS. Kirill: "Test Suite must contains Test Cases for all optionnal behaviors and when the Test Case is here must be identified per se". Is it what we want? Dimitris: Do we want TS with all optional behaviors or only with not optional behaviors. If information is in the Spec about specific behaviors you have to make it clear in the spec also. Kirill: 1st you have to determine what's optional/required, and with that in 2nd, write the TS. Dimitris: yes Lofton: I do not believe the rules of TestGL should assume a good specification. Specifically, Mark said 'the list of optional behaviors should be in the spec'. If it isn't there, the spec may be considered 'bad', but you still should be able to build a good TS. You have to make the list, as the checkpoint says. Dimitris: it is important to come up with a way to make the test suite be able to test an implementation but also report that the implementation has not failed in the case that it passes all required behaviours but fails on one or more of the optional 4. Ck 3.1 and 3.2. Are these ckpts verifiable now? Kirill: Do we still have an issue? or not? any concerns? Lofton: I have trouble to understand them. So it's hard to answer to the issue. 3.2 is quite clear. Kirill: you don't understand Test Area or testing approaches? Lofton: Test area. Kirill: The goal was to bring the modularity into the TS. To bring all dimension of variability inside the TS. In XML Schema, if you have datatype as a test Area, you will have the same Test Suite Area. KD: Is it to model the TS on the SPec? Kirill: Yes, the idea is that it's to structure the TS with the same structure of the Spec. Lofton: In some cases it will be absolutely essential. For examples profiles, if the conformance is defined in terms of profiles, so the TS will have to reflect that. The TS could be organized as areas wrt to features or profiles. Is it what you mean? Kirill: What's the structure you apply to Test Suite. One way to do it is to do it with regards to the Spec. KD: Should we ask for a declaration of how the TS is structured more than impose it. Kirill: That's a good point. Dimitris: Various technology have various needs... but we don't want to make it too free too at the same time. We don"t want to impose too muchon authors but we don't want to loose our deliverables. Kirill: Gd2 is all about how much it's important to categorize your TS. Let's leave to editors to propose a better wording for these CKs. Lofton: agreed. You can enumerate as possible choices the different possibility of organizing. KD: should we define a minimal requirement like the chapters in the spec? Dimitris: it could be dangerous. And it could be difficult to make it implementable for authors. but the list of possible organization is good. dangerous = difference between a document and a piece of software (specification vs. test suite) Lofton: the TS has to follow the technical organisation of the specifcation (modules, profiles, etc) Kirill: for a same documentn you can have multiple views. Dimitris: If you have spec which is organized in particular way it must be at least the raw model for the organization of the TS. Kirill: I agree. KD: Maybe it's just a question of drivers to address structure? Lofton: if structure is there, yes it is a question of drivers and addressing i think if structure has a direct correspondence in conformance (or conformance claims), then I think TS *must* reflect structure. Dimitris: yes, it shoud read: if you find above (in the list of alternativeorganisational manners we have written down) use it; otherwise document your TS structure and send it to QA WG so that the dec can be uptadet Kirill: We should enumerate all of those and you choose one of those. Dimitris: please document what you do in your TS and how you address the structure. Kirill: Is there any issues I'm not aware about that you want to bring to editors before the F2F QA Lofton: Any topics for next telcon. other things ? Adjourned -- Karl Dubost / W3C - Conformance Manager http://www.w3.org/QA/ --- Be Strict To Be Cool! ---
Received on Thursday, 17 October 2002 02:26:31 UTC