Re: Conformance requirement markup and test metadata (take 2)

On 9/5/11 3:22 PM, "Linss, Peter" <peter.linss@hp.com> wrote:

> On Sep 5, 2011, at 2:43 PM, Alan Stearns wrote:
> 
>> On 9/5/11 11:54 AM, "Chris Lilley" <chris@w3.org> wrote:
>> 
>>> On Friday, September 2, 2011, 4:42:46 AM, Peter wrote:
>>> 
>>> LP> On Sep 1, 2011, at 4:49 PM, Alan Stearns wrote:
>>>>> Test files currently have a reference to the section of the spec they
>>>>> test.
>>>>> We'd like to be a bit more specific about what each test covers.
>>> 
>>> I agree that this is valuable.
>>> 
>>>>> I will be
>>>>> adding metadata to Regions tests that includes any relevant id in the
>>>>> spec.
>>>>> This might just be the id of the section the test already refers to. Or it
>>>>> could be more specific, perhaps referring to one or more <dfn> ids.
>>> 
>>> LP> Currently tests have <link rel='help' href='©' /> links that
>>> LP> _must_ point to a specification section heading (i.e.: a URL from the
>>> spec's TOC).
>>> 
>>> It would seem easier to relax that restriction and let it point to a more
>>> specific phrase (ideally, a single testable assertion).
>> 
>> The way I think about it, a specification does not usually contain testable
>> assertions. Specifications contain conformance requirements that lead to
>> testable assertions. And it's not a one-to-one mapping of requirement to
>> assertion. A single requirement in a spec can lead to multiple tests, and in
>> most cases a single testable assertion is informed by multiple requirements.
>> 
>> So the metadata I'm adding to each test will likely contain more than one
>> spec id per test, which would break how the section metadata is currently
>> used. The section metadata is for collecting and reporting results, and
>> there should be only one place to report each result. The metadata I'm
>> adding is for tracking test coverage, and it makes sense in this context to
>> point back to every part of the spec that's relevant to the test.
> 
> The two aspects there (linking tests to testable assertions and linking test
> to specification sections for organization and reporting) are not mutually
> exclusive.
> 
> Currently many tests link to multiple sections within a spec. The first such
> link is considered the 'primary' and binds the test to that section of the
> spec, but it is also bound to the any other linked sections as well.
> 
> In the harness, when testing by spec section, you'll run all tests that are
> bound to that section, whether primary or secondary. When generating reports
> by section, all tests linked to each section will be listed (which causes
> tests bound to more than one section to be listed multiple times in the
> report, each test is only considered once for statistical purposes), the when
> a test has a primary link to the listed section, the test name is displayed in
> bold.
> 
> It's entirely feasible to allow the tests to simply link to other areas of the
> spec. The harness (and Shepherd) import tools will need to do a deeper search
> of the specs to find all these link points and determine which section they're
> in. Right now these tools only look at the spec's table of contents. So long
> as the link points are determinable via markup it's a manageable problem.

If you're OK with dual-purposing the specification links, then I'm happy
with using the existing format instead of inventing something new. Just to
be clear - the primary specification link could be any anchor in the spec,
and the tool would determine the enclosing section. Is that correct?

Received on Monday, 5 September 2011 22:56:35 UTC