Re: Conformance requirement markup and test metadata (take 2)

On 9/5/11 11:54 AM, "Chris Lilley" <chris@w3.org> wrote:

> On Friday, September 2, 2011, 4:42:46 AM, Peter wrote:
> 
> LP> On Sep 1, 2011, at 4:49 PM, Alan Stearns wrote:
>>> Test files currently have a reference to the section of the spec they test.
>>> We'd like to be a bit more specific about what each test covers.
> 
> I agree that this is valuable.
> 
>>>  I will be
>>> adding metadata to Regions tests that includes any relevant id in the spec.
>>> This might just be the id of the section the test already refers to. Or it
>>> could be more specific, perhaps referring to one or more <dfn> ids.
> 
> LP> Currently tests have <link rel='help' href='Š' /> links that
> LP> _must_ point to a specification section heading (i.e.: a URL from the
> spec's TOC).
> 
> It would seem easier to relax that restriction and let it point to a more
> specific phrase (ideally, a single testable assertion).

The way I think about it, a specification does not usually contain testable
assertions. Specifications contain conformance requirements that lead to
testable assertions. And it's not a one-to-one mapping of requirement to
assertion. A single requirement in a spec can lead to multiple tests, and in
most cases a single testable assertion is informed by multiple requirements.

So the metadata I'm adding to each test will likely contain more than one
spec id per test, which would break how the section metadata is currently
used. The section metadata is for collecting and reporting results, and
there should be only one place to report each result. The metadata I'm
adding is for tracking test coverage, and it makes sense in this context to
point back to every part of the spec that's relevant to the test.

Received on Monday, 5 September 2011 21:44:27 UTC