Re: Spec links in reference files

On Apr 25, 2012, at 3:54 PM, fantasai wrote:

> On 04/25/2012 02:28 PM, Rebecca Hauck wrote:
>> Hi,
>> 
>> I've recently pushed a bunch of tests following the guidelines on the wiki: http://wiki.csswg.org/test/reftest
>> 
>> Since I used the sample code there as a template for my reference file, I did not include a link to the spec I'm testing (and
>> the wiki didn't say it was required). However, leaving this out prevents us doing a filtered search for our reference files
>> for our spec on Shepherd. When I search for my reference files, I get zero results.
>> 
>> Should spec links be required (or at least recommended)? If so, does anyone object to me making the update on the wiki?
> 
> This should be filed as a bug in Shepherd, not in the format;
> if a test belonging to a test suite uses a reference file,
> then that reference file should be considered part of the
> test suite.
> 
> If we added rel=help links to a reference file, they'd be
> about what technology was used to build the reference...
> which is very rarely what you'd want to search on!
> 
> (Note: The build system already handles this.)

I don't consider this a bug in Shepherd, the search is working as designed. 

The issue here is that if you specify a specification in the search field, it searches for assets linked to that specification. As we're all aware, references are generally not linked to a specification. 

Note that is _is_ possible to link a reference to a specification, but this is an error. As Fantasai points out, that's not how the specification links are supposed to be used. Specifying a specification while searching for references will find references that are mistakenly linked to that specification.

References are only indirectly linked to specifications, in that they are linked *to* by tests that are linked in turn to a specification. We have a similar situation with support files (although the test to support file relationship is not yet tracked by Shepherd or the build system).

I've had this conversation before with Gérard off list. I can make Shepherd search on the indirect specification relationships, but then we lose the ability to search for the error state directly. We could still search for that by searching for the error message generated by the system, but it would also catch historical errors that have since been corrected, i.e.: http://test.csswg.org/shepherd/search/reference/commenter/system/comment/Reference%20must%20not%20have%20specification/

although if you also add the 'Needs Work' status it narrows it down (but you still catch historical specification links that now need work for another reason):
http://test.csswg.org/shepherd/search/reference/commenter/system/status/issue/comment/Reference%20must%20not%20have%20specification/


Since this is the second time it's come up, and people do seem to have the expectation that references used by tests bound to a specification should be searchable by that specification, I'm willing to make the change. But I want to make sure everyone understands how the system really works and what the tradeoffs are. 

Note that there are currently 16 ways to search to testing assets, as Gérard already pointed out, Rebecca could have searched for reference files in which she's listed as author:
http://test.csswg.org/shepherd/search/reference/author/rhauck/

Reference files she owns:
http://test.csswg.org/shepherd/search/reference/owner/rhauck/

Or reference files she's modified (although that's currently also catching merge changesets, I'm working on that…):
http://test.csswg.org/shepherd/search/reference/reviser/rhauck/

Peter

Received on Thursday, 26 April 2012 16:37:47 UTC