Re: Conformance requirement markup and test metadata (take 2)

On Sep 5, 2011, at 2:43 PM, Alan Stearns wrote:

> On 9/5/11 11:54 AM, "Chris Lilley" <chris@w3.org> wrote:
> 
>> On Friday, September 2, 2011, 4:42:46 AM, Peter wrote:
>> 
>> LP> On Sep 1, 2011, at 4:49 PM, Alan Stearns wrote:
>>>> Test files currently have a reference to the section of the spec they test.
>>>> We'd like to be a bit more specific about what each test covers.
>> 
>> I agree that this is valuable.
>> 
>>>> I will be
>>>> adding metadata to Regions tests that includes any relevant id in the spec.
>>>> This might just be the id of the section the test already refers to. Or it
>>>> could be more specific, perhaps referring to one or more <dfn> ids.
>> 
>> LP> Currently tests have <link rel='help' href='Š' /> links that
>> LP> _must_ point to a specification section heading (i.e.: a URL from the
>> spec's TOC).
>> 
>> It would seem easier to relax that restriction and let it point to a more
>> specific phrase (ideally, a single testable assertion).
> 
> The way I think about it, a specification does not usually contain testable
> assertions. Specifications contain conformance requirements that lead to
> testable assertions. And it's not a one-to-one mapping of requirement to
> assertion. A single requirement in a spec can lead to multiple tests, and in
> most cases a single testable assertion is informed by multiple requirements.
> 
> So the metadata I'm adding to each test will likely contain more than one
> spec id per test, which would break how the section metadata is currently
> used. The section metadata is for collecting and reporting results, and
> there should be only one place to report each result. The metadata I'm
> adding is for tracking test coverage, and it makes sense in this context to
> point back to every part of the spec that's relevant to the test.

The two aspects there (linking tests to testable assertions and linking test to specification sections for organization and reporting) are not mutually exclusive.

Currently many tests link to multiple sections within a spec. The first such link is considered the 'primary' and binds the test to that section of the spec, but it is also bound to the any other linked sections as well.

In the harness, when testing by spec section, you'll run all tests that are bound to that section, whether primary or secondary. When generating reports by section, all tests linked to each section will be listed (which causes tests bound to more than one section to be listed multiple times in the report, each test is only considered once for statistical purposes), the when a test has a primary link to the listed section, the test name is displayed in bold.

It's entirely feasible to allow the tests to simply link to other areas of the spec. The harness (and Shepherd) import tools will need to do a deeper search of the specs to find all these link points and determine which section they're in. Right now these tools only look at the spec's table of contents. So long as the link points are determinable via markup it's a manageable problem.

Received on Monday, 5 September 2011 22:25:11 UTC