Re: Conformance requirement markup and test metadata (take 2)

On Sep 5, 2011, at 3:56 PM, Alan Stearns wrote:

> On 9/5/11 3:22 PM, "Linss, Peter" <peter.linss@hp.com> wrote:
> 
>> The two aspects there (linking tests to testable assertions and linking test
>> to specification sections for organization and reporting) are not mutually
>> exclusive.
>> 
>> Currently many tests link to multiple sections within a spec. The first such
>> link is considered the 'primary' and binds the test to that section of the
>> spec, but it is also bound to the any other linked sections as well.
>> 
>> In the harness, when testing by spec section, you'll run all tests that are
>> bound to that section, whether primary or secondary. When generating reports
>> by section, all tests linked to each section will be listed (which causes
>> tests bound to more than one section to be listed multiple times in the
>> report, each test is only considered once for statistical purposes), the when
>> a test has a primary link to the listed section, the test name is displayed in
>> bold.
>> 
>> It's entirely feasible to allow the tests to simply link to other areas of the
>> spec. The harness (and Shepherd) import tools will need to do a deeper search
>> of the specs to find all these link points and determine which section they're
>> in. Right now these tools only look at the spec's table of contents. So long
>> as the link points are determinable via markup it's a manageable problem.
> 
> If you're OK with dual-purposing the specification links, then I'm happy
> with using the existing format instead of inventing something new. Just to
> be clear - the primary specification link could be any anchor in the spec,
> and the tool would determine the enclosing section. Is that correct?

Any anchor in principle, yes. I'd rather not have to track every single ID in every spec, so I want to limit the link targets to a reasonable deterministic set.

I'm not sure exactly where your conversations with Elika wound up, I presume we're looking at section anchors plus IDs on elements that also have a particular class? (or one of a set of classes, or all except those with certain classes?) Whatever we determine to be the indicator that there's something testable that needs at least one test, that can be our set of acceptable link targets.

Shepherd will be able to help ensure that all <link rel='help'> href's point to a known target (it produces a warning on push and files an issue against the test if there are links to unknown targets). Here's an example:
http://test.csswg.org/shepherd/testcase/writing-mode-vertical-lr-002/status/issue/
(the link is to the property definition.)

Right now it only knows sections. So for sanity's sake, let say that tests can only point to targets other than section headings when the spec has the additional conformance markup to determine what the additional legal targets are…  Let me know the rules to determine the actual link targets and I'll build the tool to extract them.

Peter

Received on Tuesday, 6 September 2011 01:31:32 UTC