Re: Linking test suites to specs

Hi Gérard,

Thanks for your input on this! Comments inline below...

On 6/25/13 10:23 AM, ""Gérard Talbot"" <css21testsuite@gtalbot.org> wrote:

>
>Le Lun 24 juin 2013 19:30, Rebecca Hauck a écrit :
>> Hi all,
>>
>> Some of the CSS specs have a "Test Suite" in the header, which I presume
>> was part of an original spec template. However, in the specs I've come
>> across that have it, it's either not used/updated [1][2][3], or it's
>> used inconsistently [4][5][6].
>>
>> I was talking to fantasai about this a while ago (in the context of
>> defining Test Owners at the spec level) and she suggested we link to the
>> test suite cover page that is part of the nightly build. For example,
>> [7].
>>
>> In an effort to make this consistent across all the the CSS specs and
>> for the clearest test suite visibility (even so we can clearly see when
>> there are none), I'm proposing the following:
>>
>>  1.  Add the Test Suite section to all of the specs ­ those with no
>> tests labeled "none yet"
>>  2.  Link to the test suite cover page per fantasai's suggestion
>>
>> If this is put into place, I'll then be modifying & using the test suite
>> cover page to include things like test owner(s), coverage data, tests
>> needed, etc.
>>
>> Any objections?
>>
>> May I add this to the agenda for this week's meeting? Once I have
>> consensus, I'll make the updates accordingly.
>>
>> Thanks,
>> -Rebecca
>>
>> [1] http://www.w3.org/TR/css3-transforms/ - "none yet"
>> [2] http://www.w3.org/TR/css3-transitions/ - "none yet"
>> [3] http://www.w3.org/TR/css-overflow-3/ - "none yet"
>> [4] http://www.w3.org/TR/css3-conditional/ - separate links to shepherd
>> and the harness (suite interface)
>> [5] http://www.w3.org/TR/css-masking/ - links to the harness (runner)
>> [6] http://www.w3.org/TR/filter-effects/ - links to the harness (runner)
>> [7] http://test.csswg.org/suites/css3-transforms/nightly-unstable/
>
>
>Rebecca,
>
>
>>  1.  Add the Test Suite section to all of the specs ­ those with no
>> tests labeled "none yet"
>
>Just a thought here. Why not add a Test Suite section and a Take the
>Test Harness section? Some will prefer the Test Suite, others taking
>some tests in the Test Harness.

As Peter mentioned in a subsequent response to yours, the harness is
already linked in the test annotation widgets (albeit on only the editor's
drafts). He and I have added those annotations to a number of specs
recently and will catch up with the rest missing them soon. (There
shouldn't be that many more left at this point).  Keep in mind that I do
intend to redesign that test suite landing page to include many more
useful things than it currently has, so the harness link will definitely
be there. Of course, I'll solicit feedback from you and the rest of the
group when I modify that page template.

Also, the main point of this thread was to simply establish a clear and
consistent link between each spec and its test suite. I personally don't
have any objection to adding two links to the spec header instead of one,
but unnecessary or redundant clutter may be concerns that other
stakeholders have.

>
>There is also a few other issues.
>
>- A majority of tests have not been reviewed at all. There are tests
>that are not precise and that are not correct out there. The spec maybe
>in CR while its test suite status maybe less, much less mature, much
>less reliable. There ought to be some indication of this prior to taking
>tests.

Excellent point. I can summarize this on the test suite landing page. I'll
work with Peter to see if I'd be able leverage a the Shepherd API here.


>- There are CSS2.1 tests that have comments, explanations with
>corrections available and still have not been corrected yet.

This should also be addressed if I can leverage the Shepherd API as I
presume all such tests are flagged there.


>- Some browsers, like Firefox (but not restricted to Firefox), require
>to set some setting in order to try a particular test suite: eg.
>layout.css.supports-rule.enabled must be enabled in order to try the CSS
>Conditional Rules Test suite. Otherwise, the test suite for such browser
>is useless.

I'm not sure how to respond to this in the context of the test suite page
being linked to the specs. Hows is this requirement communicated now in
Shepherd or the harness? And is a vendor-specific requirement really
something that should be communicated by the W3C anyway? I'm inclined to
think that the onus would be on the vendors actually running the tests to
know what they're requirements are, but I'm open to your ideas on how this
should be addressed here.


>If Test Suites and tests become more visible and widely available, it
>may - in fact, I'm sure it will - easily lead to false claims, careless
>exaggerations, premature claims about one browser over another.

Again, I'm not sure what you're suggesting here. Are you saying that the
suites should _not_ be made more visible? Or just that we need to be
careful to accurately describe what's in the test suite re: approved/not
approved?  If the former, I very much disagree. In fact, I'd argue that
the lower visibility a test suite has to the broader public, the more
discrepant claims arise as typically (and historically) vendors develop
their own test suites, which inevitably differ from the others. Perhaps
I'm being simplistic, but the more visible a test suite is, the more
implementors use it, which is sort of the point of having it in the first
place.

Further, it's very easy to find any given W3C spec, but often not as
simple to find its tests -- it usually takes a couple more searches and a
few more clicks, if you're savvy. Back to the simple point of this thread,
the test suites should be consumed along with the specs and we should
leverage the high discoverability of the specs to make the test suite just
as accessible.  I definitely agree with you though that we need to be
clear on the state of test suites.

-Rebecca

Received on Wednesday, 26 June 2013 00:24:19 UTC