W3C home > Mailing lists > Public > www-qa@w3.org > June 2003

Re: Should Test Assertions be required

From: Lofton Henderson <lofton@rockynet.com>
Date: Mon, 09 Jun 2003 08:50:49 -0600
Message-Id: <5.1.0.14.2.20030609084445.027fda40@rockynet.com>
To: Mark Skall <mark.skall@nist.gov>
Cc: www-qa@w3.org

At 10:37 AM 6/9/03 -0400, Mark Skall wrote:

>I think the real question is being missed.  It is:  "Who should develop 
>the test assertions - the spec developers or the test suite 
>developers?"  Once we decide who this should be, we'll have our answer.

Not necessarily.  The current sense of GL14 is that a set of test 
assertions needs to be available for and associated with each Rec.

Elsewhere we ask the WG to commit that some test materials will also be 
associated with each Rec, so the timing is such that any TAs would be 
available in the right time frame, regardless of who writes them.

So ... if we were to answer that TS developers should provide them, does 
that mean that GL14 and its CPs go away?  Or does it just mean that someone 
else does the labor of generating the TAs?


>There are pros and cons for each of these, as well.

I agree.

-Lofton.




>At 12:46 PM 6/6/2003 -0400, david_marston@us.ibm.com wrote:
>
>>Lynne Rosenthal instiagted this thread by writing:
>> >Should specification developers be required to include Test Assertions
>> >(TA) in their specifications?
>> >some Pros:
>> >1. ensures testable specifications
>> >2. consensus in what is required
>> >3. facilitates test generation
>>
>>To which I would add:
>>4. useful for precise citations (by test cases, emails discussing fine
>>    points, etc.)
>>
>> >some Cons:
>> >1. cost (time) to create
>> >2. may not be appropriate or adequate for test developer's use...
>> >3. not needed since generate tests directly from Spec....
>>[#3 is actually from a schema of the markup language, for those specs
>>that define an XML vocabulary.]
>>
>> >IMO specifications should (must?) identify the testable statements or
>> >test requirements.
>>
>>To me, the above is the very epitome of a Priority 2 requirement. It's
>>a substantial benefit to have them, not just "nice to have" but also not
>>vital to the task of writing test cases. Right now, we manage to isolate
>>testable passages in the current specs.
>>
>> >But depending on your definition of TA, these may not be as formally
>> >represented (stated) as a TA.
>>
>>Strongly agree. Given the current spec-writing practices, even simple
>>tags surrounding testable sentences ("virtual highlighter") would fill
>>a lot of the needs. Some spec editors (and WG members who get cajoled
>>into writing substantial amounts of verbiage) may not buy in to having
>>separate passages full of TAs, but they may be willing to put tags
>>around the sentences that they were writing anyway.
>>
>>I think that the W3C's spec-writing practices will improve over time.
>>The 1.0 Spec Guidelines need to address the weaknesses that allow
>>unintended varying interpretations (see Pro#2 above) and related
>>threats to interoperability. Most of the Pros and Cons listed here
>>are not exclusive to TAs; we can achieve better testability and
>>consensus without TAS; but each technique that contributes to the goals
>>can be measured in a cost/benefit way relative to the other techniques.
>>The 2.0 Spec Guidelines can push for a higher level, possibly even
>>building on new designs for machine-processable assertions. (I have no
>>guess about a year when 2.0 might be necessary.)
>>
>>Overall, I would like to encourage spec editors to mark up passages
>>that have the level of testability that TAs have. If they don't want to
>>have visible distinctiveness for such passages, that's fine with me for
>>the time being. I hope that some of the specs will be more adventurous.
>>.................David Marston
>
>****************************************************************
>Mark Skall
>Chief, Software Diagnostics and Conformance Testing Division
>Information Technology Laboratory
>National Institute of Standards and Technology (NIST)
>100 Bureau Drive, Stop 8970
>Gaithersburg, MD 20899-8970
>
>Voice: 301-975-3262
>Fax:   301-590-9174
>Email: skall@nist.gov
>****************************************************************
>
Received on Monday, 9 June 2003 10:50:10 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:14:00 GMT