W3C home > Mailing lists > Public > www-qa-wg@w3.org > June 2003

Draft Minutes QAWG Telcon Monday 09 June 2003

From: Andrew Thackrah <andrew@opengroup.org>
Date: Wed, 11 Jun 2003 14:39:42 +0100 (BST)
To: <www-qa-wg@w3.org>
Message-ID: <Pine.LNX.4.33.0306111437170.14403-100000@hyperion>


DRAFT MINUTES

QA Working Group Teleconference
Monday, 09-June-2003
--
Scribe: Andrew Thackrah

Attendees:
(PC) Patrick Curran (Sun Microsystems)
(KD) Karl Dubost (W3C, WG co-chair)
(PF) Peter Fawcett (RealNetworks)
(KG) Kirill Gavrylyuk (Microsoft)
(LH) Lofton Henderson (CGMO - WG co-chair)
(LR) Lynne Rosenthal (NIST - IG co-chair)
(MS) Mark Skall (NIST)
(AT) Andrew Thackrah (Open Group)
(DM) David Marston (IBM, guest)

Regrets:
(KD) Karl Dubost (W3C, WG co-chair)
(DH) Dominique HazaŽl-Massieux (W3C)
(SM) Sandra Martinez (NIST)

Absent:
(DD) Dimitris Dimitriadis (Ontologicon)



Summary of New Action Items:

 [no action items were assigned in this meeting]

Agenda:
<a href="http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0019.html">
http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0019.html</a>

Previous Telcon Minutes:
<a href="http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0010.html">
http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0010.html</a>


Minutes:

Agenda
1. Roll call, 1100 EDT
1b. Crete F2F attendance
2. Last Call reviews
3. SpecGL issue processing
4. specGL extras

1. Roll call

	See above

1b. Crete face-to-face meeting atendance
	[This ad oc item was added during the meeting.]

	Daniel Dardailler will only be present for two day (Mon, Tue)
	All W3C team shoul dbe present

	LH, PC, MS, (and DD??) confirmed attendance. AT said yes, subject to travel approval.

2 last call reviews

	[none]

3 SpecGL issue processing

	ref2a from agenda:

	LR summarises: purpose is to stimulate discusion re. test assertions. e.g. definition of a
	Test Assertion (TA). Should SpecGL require TAs to be included? How formal should TAs be?


	LH - do we still have any fundamental confusion about TA definition?
	Seattle resolution added GL14 definition to the glossary.

	[LH reads this definition]

	MS - Is there a link to this definition?

	LH - Agenda link [1] then from TOC jump to GL14. Also in agenda is the ref to AT's explanation
	of the difference between a conformance requirement (CR) and an assertion and MS's example exercise.

	I believe a spec could be written with embedded TAs - but doesn't have to be that way. See my 3
	examples in mail [ref]

	MS - difference of opinion. conformance requirements != assertions. We need to clarify this.

	LH - consider a spec with a formal notation: TAs are irrelevant because tests can be auto-generated

	LR - MUSTs etc are important and should be ID'd or tagged - it is not necessary to have TAs.
	Spec authors need to concentrate on the identification of CR that will result in TAs

	LH - SpecGL is an an example - not written with TAs

	MS - My example suggests that SpecGL is not not exact

	LH - SpecGL should be considered a reasonable way to write a spec

	MS - consider the issue of different people producing different TAs from the same document

	DM - an aside - even with TAs incuded - it will still be important to identify testable prose - ie
	we need tags for prose regardless of TAs

	KG - So what is the difference betweem TA & CRs?

	LH - agenda ref 2c - long discussion of this from AT. there was general agreement on this
	definition

	PC - there may be a many-to-one relationship between TA & CR... small CRs, large # TAs.
	For example- Java api specs: total number of TAs is 5 figures (tens of thousands). I want
	a spec to have assertions tagged. But there are only a small number of CRs - all high level.

	AT - are all TAs published?

	PC - yes, if possible. non-contiguous fragments of test may make up assertions, which is why it
	is important to be able to relate every TA back to spec using tags etc.

	LH - We had similar experience for SVG detailed tests.

	PC - So I think we should encourage (inclusion of) TAs where appropriate but not mandate.

	LR, LH, AT - agree


	MS - There are still two different things, logically: do we require spec authors to develop TSs
	when possible or leave it to test developers?

	PC - I'm in favour of encouragment, not mandates. In perfect world - spec developers would do it.

	MS - If the CRs are only in spec author's mind - then their TAs will be biased, wheres test developer's
	TAs will be objective

	DM - BNF/schema are sufficiently formal to be considered as good as TAs

	[group decides to ignore specs with formal grammars for the moment]

	PC - The fuzzier the language, the less useful. tagging testability helps remove subjectivity

	MS - But a TA must be mapped to normative spec language

	LH - The job of who does the work is irrelevant - so long as TAs are published in some form?

	MS - we should confirm who we want TAs to come from - test developers or spec authors

	KG - If TAs are written by spec authors - quality may be low. we use TAs a mechanism for spec reviews.
	So we have someone other than spec author make TAs - as a QA on the spec.

	[general agreement on value of this]

	LH - Should we require TAs by time we get to Rec.?

	KG - If it's priority 2 , then yes, desirable

	DM - It's a priority 3 to me.

	PC - Test case is useless without assertion

	[noisy discussion follows. Conclusion is that it is desirable (essential to some) to be able to
	trace an assertion back to a specific section (or sections) of spec. text. TA is link between test and
	spec. Test has no validity without this link.]

	LH - So TAs, ideally not created by spec author, give positive benfit to quality of spec. right?

	MS - Quality yes, also one set of TAs creates consistency of testing.

	LH - straw poll - keep GL 14?

 	[all YES in principle]

	Three spin-off issues are identified that should be addressed in future meetings:

 	1 Priority
 	2 Creator (spec author | test developer )
 	3 Auto-generation (spec->test via formal grammar)


	[argument about relationship between normative text and TAs follows]

	LR - There is the danger of setting the bar too high for spec writers to rephrase text as an assertion.

	[discussion follows regarding clarity of spec language ]

	MS - need to avoid words like obvious, clear. [referring to sugestions that TAs may be explicit if
	spec CRs are sufficiently clear. Considered too subjective].

	AT - should TAs be mandatory?

	LH - issue 98 - decided at Tokyo.

	[There is discussion about who is mandated - test developers or product implementers]

	PC - We recognize that our tests can not give 100% coverage. We have a conformance clause -
	obliged to implement regardless.

	DM - Also need to handle bad assertions

	PC - TAs should not be mandatory for implementers - only for test developers.

	[There is some agreement that test developers should be mandated to implement any testable assertions
	but that it is undesirable for product implement to use TAs - they should implement directly from
	the specification conformance requirements]


	[summing up]
	LH - please give opionions on remainder of issues in [2a] in advance of crete.


	[agenda items 4+ not addressed]


Adjourned 12:00
Received on Wednesday, 11 June 2003 09:39:48 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:13 GMT