Final Minutes of QAWG Telcon Monday 09 June 2003

QA Working Group Teleconference
Monday, 09-June-2003
--
Scribe: Andrew Thackrah

Attendees:
(PC) Patrick Curran (Sun Microsystems)
(KD) Karl Dubost (W3C, WG co-chair)
(PF) Peter Fawcett (RealNetworks)
(KG) Kirill Gavrylyuk (Microsoft)
(LH) Lofton Henderson (CGMO - WG co-chair)
(LR) Lynne Rosenthal (NIST - IG co-chair)
(MS) Mark Skall (NIST)
(AT) Andrew Thackrah (The Open Group)
(DM) David Marston (IBM, guest)

Regrets:
(KD) Karl Dubost (W3C, WG co-chair)
(DH) Dominique Hazaël-Massieux (W3C)
(SM) Sandra Martinez (NIST)

Absent:
(DD) Dimitris Dimitriadis (Ontologicon)



Summary of New Action Items:

  [no action items were assigned in this meeting]

Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0019.html

Previous Telcon Minutes: 
http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0010.html


Minutes:

Agenda
1. Roll call, 1100 EDT
1b. Crete F2F attendance
2. Last Call reviews
3. SpecGL issue processing
4. specGL extras

1. Roll call

	See above

1b. Crete face-to-face meeting atendance
	[This ad oc item was added during the meeting.]
	 
	Daniel Dardailler will only be present for two day (Mon, Tue)
	All W3C team shoul dbe present
	 
	LH, PC, MS, (and DD??) confirmed attendance. AT said yes, subject 
to travel approval.

2 last call reviews
	 
	[none]

3 SpecGL issue processing 
	ref2a from agenda:

	LR summarises: purpose is to stimulate discusion re. test 
assertions. e.g. definition of a 	Test Assertion (TA). Should SpecGL 
require TAs to be included? How formal should TAs be?


	LH - do we still have any fundamental confusion about TA 
definition? 	Seattle resolution added GL14 definition to the glossary.

	[LH reads this definition]

	MS - Is there a link to this definition?

	LH - Agenda link [1] then from TOC jump to GL14. Also in agenda is 
the ref to AT's explanation
	of the difference between a conformance requirement (CR) and an 
assertion and MS's example exercise. 	 	I believe a spec could be 
written with embedded TAs - but doesn't have to be that way. See my 3 	 
examples in earlier mail.

	MS - difference of opinion. conformance requirements != 
assertions. We need to clarify this.

	LH - consider a spec with a formal notation: TAs are irrelevant 
because tests can be auto-generated

	LR - MUSTs etc are important and should be ID'd or tagged - it is 
not necessary to have TAs. 	Spec authors need to concentrate on the 
identification of CR that will result in TAs
	 
	LH - SpecGL is an an example - not written with TAs

	MS - My example suggests that SpecGL is not not exact

	LH - SpecGL should be considered a reasonable way to write a spec

	MS - consider the issue of different people producing different 
TAs from the same document

	DM - an aside - even with TAs incuded - it will still be important 
to identify testable prose - ie 	we need tags for prose regardless 
of TAs

	KG - So what is the difference betweem TA & CRs?

	LH - agenda ref 2c - long discussion of this from AT. there was 
general agreement on this 	definition
	 
	PC - there may be a many-to-one relationship between TA & CR... 
small CRs, large # TAs. 	For example- Java api specs: total number 
of TAs is 5 figures (tens of thousands). I want 	a spec to have 
assertions tagged. But there are only a small number of CRs - all high 
level.
	 
	AT - are all TAs published?

	PC - yes, if possible. non-contiguous fragments of test may make 
up assertions, which is why it 	is important to be able to relate every TA 
back to spec using tags etc.

	LH - We had similar experience for SVG detailed tests.

	PC - So I think we should encourage (inclusion of) TAs where 
appropriate but not mandate.
	 
	LR, LH, AT - agree


	MS - There are still two different things, logically: do we 
require spec authors to develop TSs 	when possible or leave it to test 
developers?
	 
	PC - I'm in favour of encouragment, not mandates. In perfect world 
- spec developers would do it.

	MS - If the CRs are only in spec author's mind - then their TAs 
will be biased, wheres test developer's
	TAs will be objective
	 
	DM - BNF/schema are sufficiently formal to be considered as good 
as TAs
	 
	[group decides to ignore specs with formal grammars for the moment]

	PC - The fuzzier the language, the less useful. tagging 
testability helps remove subjectivity

	MS - But a TA must be mapped to normative spec language

	LH - The job of who does the work is irrelevant - so long as TAs 
are published in some form?

	MS - we should confirm who we want TAs to come from - test 
developers or spec authors	 

	KG - If TAs are written by spec authors - quality may be low. we 
use TAs a mechanism for spec reviews.
	So we have someone other than spec author make TAs - as a QA on 
the spec.
	 
	[general agreement on value of this]
	 
	LH - Should we require TAs by time we get to Rec.?

	KG - If it's priority 2 , then yes, desirable

	DM - It's a priority 3 to me. 
	PC - Test case is useless without assertion 	 
	[noisy discussion follows. Conclusion is that it is desirable 
(essential to some) to be able to 	trace an assertion back to a 
specific section (or sections) of spec. text. TA is link between test 
and 	spec. Test has no validity without this link.]

	LH - So TAs, ideally not created by spec author, give positive 
benfit to quality of spec. right?

	MS - Quality yes, also one set of TAs creates consistency of 
testing.

	LH - straw poll - keep GL 14?

  	[all YES in principle]
  	Three spin-off issues are identified that should be addressed in 
future meetings:
   	1 Priority
  	2 Creator (spec author | test developer )
  	3 Auto-generation (spec->test via formal grammar)
  
	[argument about relationship between normative text and TAs 
follows]

	LR - There is the danger of setting the bar too high for spec 
writers to rephrase text as an assertion.
	 
	[discussion follows regarding clarity of spec language ]	 
	 
	MS - need to avoid words like obvious, clear. [referring to 
sugestions that TAs may be explicit if 	spec CRs are sufficiently clear. 
Considered too subjective]. 
	AT - should TAs be mandatory?

	LH - issue 98 - decided at Tokyo.

	[There is discussion about who is mandated - test developers or 
product implementers]
	 
	PC - We recognize that our tests can not give 100% coverage. We 
have a conformance clause - 	obliged to implement regardless.

	DM - Also need to handle bad assertions

	PC - TAs should not be mandatory for implementers - only for test 
developers.
	 
	[There is some agreement that test developers should be mandated 
to implement any testable assertions 	but that it is undesirable for 
product implement to use TAs - they should implement directly from 	 
the specification conformance requirements]
	 

	[summing up]
	LH - please give opionions on remainder of issues in [2a] in 
advance of crete.


	[agenda items 4+ not addressed]

	 
Adjourned 12:00

Received on Tuesday, 1 July 2003 12:41:51 UTC