DRAFT - QA telconf 2002-09-25

Sorry,

I have just noticed that I have completely forgotten the minutes to 
the post the minutes of the last Telconf.


Lofton: Spec GL (follow-up on previous meeting, overdue agenda item).

Karl: This is an issue with Spec GL which is very crucial part of the 
QA Framework. The Spec GL are not clear enough with an external point 
of view, it's really hard to if you don't know all the discussions we 
have been having, it's really difficult to read. Who will apply the 
Spec GL on themselves?

Dom: Mark is supposed to do so

Karl: It's really important to have a new look at them. One idea 
would be to move the examples from the SpeC GL to the ExTechs. A 
lighter spec easy to use, fast to read. Let's keep in mind that 
people reading the Spec GL won't be aware of the whole discussion 
behind that. It's really boring to read them right now, having a 
better logic behind it would help to understand and make it more 
usable for editors. It seems hard to understand and too constraintful 
right now and enough inviting.

Lofton: That looks like some of the issues that Alex pointed in his 
emails. Lynne seems to have the same feelings as well. She said she 
had a hard time reading it after 3 months with no contact with QA WG. 
That's a very important point of view because she is part of the WG. 
So I think we need to have a close look at clarity and simplicity.

2 more comments: Lynne is coeditor with Dominique and her role is to 
identify specific issues in the recent mails. I hence have 21 new 
issues to integrate in the issues list. Most of them came out of the 
mail exchange with Alex and some of them matches what you're talking 
about.

Karl: I've asked someone in the Team who has never read a W3C spec to 
read the spec. The one who'll do that is an intern in W3C Team I was 
about to pull out the examples from the Spec GL and put them in the 
ExTechs doc.

Lofton: That's what I was about to suggest and then, we can look at 
Spec GL, only keeping some of them used for illustrations to remove 
ambiguity and that will open up a lot of issues esp. whether specs 
should have examples since it should apply to us as much as other 
specs.

Dom: I think we actually need to change this checkpoint as suggested 
Alex, maybe somehitng "give an example for each ambiguous req".

Lofton: we'll need to think more on this, but later next week, we'll 
have a new block of identified issues that'll help getting ready for 
tokyo. Some are editorial, some others are at the heart of the DOV 
model. Karl, your point is well taken. You should proceed and put the 
examples from Spec GL into extechs. I personnally agree with lots of 
comments you made. I think Alex made a lot of good points too.

Dimitris: We should probably contact one or 2 editors of very heavy 
specs and ask them explicitely to comment on it.

Lofton: should we ask on the current state? or wait for the next 
publication in one month?

Dimitris: let's do that after next publication but let's do this! 
I'll try to approach chairs/editors unofficially to have at least 
some comments before Tokyo


=== Test GL draft and issues ===

Kirill presents the Test GL.
http://www.w3.org/QA/WG/2002/framework-20020826/qaframe-test
http://www.w3.org/QA/WG/qawg-issues-html.html#x79
There are not so many issues right now. The most common one has been 
raised by Alex. "How to create a good test suite". For each 
checkpoints, we say what's the best way to achieve this checkpoint.

Lofton: In SpecGL, because of the imperative wording in many of the 
guidelines, it could seem that we ask for a process, more than the 
final result itself, which leads to unclarity. For this reason, we 
are sometimes misunderstood in the guidelines. In SpecGL, we do focus 
more on the results than the process, in the descriptions of the 
guidelines and checkpoints

missed KG.

Lofton: I don't necessary agree with Alex's TestGL point (issue #79). 
But I would try to clarify the spec to make it clear whether we are 
addressing process versus result in each particular guideline and 
checkpoint.  Then we can argue the issue about whether it should be 
result only, or process also.

Kirill: I understand. Ops GL and Test GL are very entighten together. 
Test GL are just a continuation of the process (Op GL).

Lofton: if we look at the checkpoints and enumerate what things must 
be part of the TS, then we could rephrase it for example to have only 
the result in the checkpoint and move the process into the ExTech

Kirill: gives an example for CK 1.1

Dimitris: We could rewrite checkpoints according to the distinction 
between how things need to be done and what must be in them , however 
editors should only investigate time if the distinction definitely 
holds

Lofton: I think there's a distinction... Can we clarify for ourselves 
the nature of process versus results... and if we have done that, we 
will be able to solve the issues. One way of looking at it is to look 
at "what it should contain" as TS deliverables

Dimitris: We must first decide what's the list of deliverables and 
with this list we will be able to write in a way that will be clear.

Kirill: no objections. I would like to just go and propose 
reformulation of the guidelines. So we can let the old text and new 
text... and choose.


Gd 4. How should we coordinate Ops and Test guidelines. Is it ok to 
duplicate checkpoints?

Kirill: we do a lot of process in the Guideline 4 right now. Maybe 
after reformulation, we will not have anymore a duplication of the 
TEst GL.
Issue 2 resolved after no objections.


Ck 1.7 Identify optional behaviors in the specification. Do we keep 
this checkpoint or drop it?

Kirill: We still haven't decide if we should skip this CK, or if it 
should go in the Spec Guidelines.

Lofton: The test Suite has to know about optional behaviors. Do we 
mean discretionary behaviors.

Kirill: No for examples you can have one binding and only one 
implementation for this particular binding.

Lofton: Optional behaviors have to be identified in the TS or TS 
harness itself. I.e., the TS/harness must be aware of whether a test 
is for an optional behavior or not, and must be aware of whether the 
implementation claims to do it or not.  There's something similar in 
the XSLT test suite.

Kirill: is it a requirements or up to the implementations of the TS.

Kirill: "Test Suite must contains Test Cases for all optionnal 
behaviors and when the Test Case is here must be identified per se". 
Is it what we want?


Dimitris: Do we want TS with all optional behaviors or only with not 
optional behaviors. If information is in the Spec about specific 
behaviors you have to make it clear in the spec also.

Kirill: 1st you have to determine what's optional/required, and with 
that in 2nd, write the TS.

Dimitris: yes

Lofton: I do not believe the rules of TestGL should assume a good 
specification. Specifically, Mark said 'the list of optional 
behaviors should be in the spec'.  If it isn't there, the spec may be 
considered 'bad', but you still should be able to build a good TS. 
You have to make the list, as the checkpoint says.

Dimitris: it is important to come up with a way to make the test 
suite be able to test an implementation but also report that the 
implementation has not failed in the case that it passes all required 
behaviours but fails on one or more of the optional

4.    Ck 3.1 and 3.2. Are these ckpts verifiable now?
Kirill:
Do we still have an issue? or not? any concerns?

Lofton: I have trouble to understand them. So it's hard to answer to 
the issue. 3.2 is quite clear.

Kirill: you don't understand Test Area or testing approaches?

Lofton: Test area.

Kirill: The goal was to bring the modularity into the TS. To bring 
all dimension of variability inside the TS. In XML Schema, if you 
have datatype as a test Area, you will have the same Test Suite Area.

KD: Is it to model the TS on the SPec?

Kirill: Yes, the idea is that it's to structure the TS with the same 
structure of the Spec.

Lofton: In some cases it will be absolutely essential. For examples 
profiles, if the conformance is defined in terms of profiles, so the 
TS will have to reflect that. The TS could be organized as areas wrt 
to features or profiles. Is it what you mean?

Kirill: What's the structure you apply to Test Suite. One way to do 
it is to do it with regards to the Spec.

KD: Should we ask for a declaration of how the TS is structured more 
than impose it.

Kirill: That's a good point.

Dimitris: Various technology have various needs... but we don't want 
to make it too free too at the same time. We don"t want to impose too 
muchon  authors but we don't want to loose our deliverables.

Kirill: Gd2 is all about how much it's important to categorize your 
TS. Let's leave to editors to propose a better wording for these CKs.

Lofton: agreed. You can enumerate as possible choices the different 
possibility of organizing.

KD: should we define a minimal requirement like the chapters in the spec?

Dimitris: it could be dangerous. And it could be difficult to make it 
implementable for authors.  but the list of possible organization is 
good.
dangerous = difference between a document and a piece of software 
(specification vs. test suite)

Lofton: the TS has to follow the technical organisation of the 
specifcation (modules, profiles, etc)

Kirill: for a same documentn you can have multiple views.

Dimitris: If you have spec which is organized in particular way it 
must be at least the raw model for the organization of the TS.

Kirill: I agree.

KD: Maybe it's just a question of drivers to address structure?

Lofton: if structure is there, yes it is a question of drivers and 
addressing i think if structure has a direct correspondence in 
conformance (or conformance claims), then I think TS *must* reflect 
structure.

Dimitris: yes, it shoud read: if you find above (in the list of 
alternativeorganisational manners we have written down) use it; 
otherwise document your TS structure and send it to QA WG so that the 
dec can be uptadet

Kirill: We should enumerate all of those and you choose one of those.

Dimitris: please document what you do in your TS and how you address 
the structure.

Kirill: Is there any issues I'm not aware about that you want to 
bring to editors before the F2F QA

Lofton: Any topics for next telcon. other things ?

Adjourned
-- 
Karl Dubost / W3C - Conformance Manager
           http://www.w3.org/QA/

      --- Be Strict To Be Cool! ---

Received on Tuesday, 1 October 2002 15:57:44 UTC