telcon minutes, 20040405, draft

All,

For you review

Best

/Dimitris

---

QA Working Group Teleconference
Monday, 5-April-2004
--
Scribe: Dimitris Dimitriadis

Attendees:
(PC) Patrick Curran (Sun Microsystems)
(DD) Dimitris Dimitriadis (Ontologicon)
(DH) Dominique Hazael-Massieux (W3C)
(LH) Lofton Henderson (CGMO - WG co-chair)
(LR) Lynne Rosenthal (NIST - IG co-chair)
(MS) Mark Skall (NIST)

Regrets:
(KD) Karl Dubost (W3C, WG co-chair) r
(MC) Martin Chamberlain (Microsoft) r
(AT) Andrew Thackrah (Open Group) r
(VV) Vanitha Venkatraman (Sun Microsystems) r

Absent:

Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2004Apr/0011.html
Previous Telcon Minutes: 
http://lists.w3.org/Archives/Public/www-qa-wg/2004Apr/0010.html

Minutes:

2.) routine business

- Future telecons [0]
[lh] note that next week's telcon is rescheduled for Wednesday

- date change for June F2F
everyone preferred the week 15, 16, 17 June

- other?

3.) TestLite
[pc] changes from the previous one: this one is not in bullet form, as 
the previous one. two of the previous
principles (about diagnostic information) were incorporated into 
current principle two (talking about repeatable and deterministic 
tests). Where possible, operational issues should be worked into the 
body of the document. Also tries to avoid heavy duty requirements. 
Tried to limit the number of requirements.

As a user of the test suite you need to know if it applies to you at 
all, also you need to understand how comprehensive it is. Most people 
want to know what the test suite tests and what not.

[lh] these are all commandments on a human agents, what orientation do 
you intend to give?
[pc] since we've agreed to incorporate process issues into this 
document, we still have the (older) problem of which is this documents 
class of product.
[dh] the principle title shouldn't use RFC keywords. we should perhaps 
use different stylistic form between normative and non-normative 
requirements.
[lh] it can be a short descriptive title. another question: are we 
going to try for the first publication how spec, test and qa handbook 
are coordinated?
[dm] desirable to show that that's the plan
[lh] so, for the first publication, speking of a unified approach?
[dd] propose to write documents separately
[lh] offline discussion about guidelines and principles, good 
practices; aim at convergence

1.2
[pc] coverage information: potentially something that can't be met. in 
previous drafts we required assertion lists, now present in terms of 
good practice. there should be something in the metadata about the 
tests, think we should require at least that (as a minimum).
[pc] shall we use the term guideline?

2.3 reproducible and repeatable
[pc] all other things equal (machines, setup etc) running the test 
suite should yield indentical results
[ms] if you have different hardware, the implementation may behave 
differently
[pc] components of test suite? seems ok, no objections
[pc] unambigous information on what tests need to be executed
[pc] document how to execute tests. in the case of a test harness, you 
allow people to write their own, which may bring test result into 
question (violates the principle of reproducibility)
[dd] either require a harness or don't mention it at all
[dm] you may at least have a mechanism that states what tets have been 
run for a particular combination of optionality
[lr] i think it's a good practice, not best practice
[pc] whatever the test result is (see below) it should be captured by 
the harness

2.4
[pc] allow test to report results
[lr] conformance testing has nothing to do with why things do not work, 
rather if it does or not. diagnostic information is a good practice at 
most.
[dm] i run the test suite, i see failures, i ask myself if i run the 
correct tests, then i look at the implementation.

3. evolution over time
[pc] multiple releases, respond to bug-reports. have to be worked into 
the document
[lh] I like operational advice/guidelines
[lr] don't see a problem introducing a fourth guideline
[lh] 1,2,3 is where the normative information resides, introducing a 
fourth guideline implies the same level of normativity

general
[pc] is the level of detail ok?
[lh] one question is as each principle gets deveoped and refined, 
should they be refined to the point where we can suggest useful tools?
[pc] that shouldbe a goal
[dd] in a good practices document
[lh] what kind of things like templates, harnesses etc can be targeted?
[lr] focus more on examples, for writing specs and test suites, don't 
worry about stories

4.) Adjourn
call adjourned at 3 minutes past the hour

5.) Overflow (12-12:30): available.

Received on Monday, 5 April 2004 12:22:04 UTC