Re: Linking test suites to specs

Le Lun 24 juin 2013 19:30, Rebecca Hauck a écrit :
> Hi all,
>
> Some of the CSS specs have a "Test Suite" in the header, which I presume
> was part of an original spec template. However, in the specs I've come
> across that have it, it's either not used/updated [1][2][3], or it's
> used inconsistently [4][5][6].
>
> I was talking to fantasai about this a while ago (in the context of
> defining Test Owners at the spec level) and she suggested we link to the
> test suite cover page that is part of the nightly build. For example,
> [7].
>
> In an effort to make this consistent across all the the CSS specs and
> for the clearest test suite visibility (even so we can clearly see when
> there are none), I'm proposing the following:
>
>  1.  Add the Test Suite section to all of the specs – those with no
> tests labeled "none yet"
>  2.  Link to the test suite cover page per fantasai's suggestion
>
> If this is put into place, I'll then be modifying & using the test suite
> cover page to include things like test owner(s), coverage data, tests
> needed, etc.
>
> Any objections?
>
> May I add this to the agenda for this week's meeting? Once I have
> consensus, I'll make the updates accordingly.
>
> Thanks,
> -Rebecca
>
> [1] http://www.w3.org/TR/css3-transforms/ - "none yet"
> [2] http://www.w3.org/TR/css3-transitions/ - "none yet"
> [3] http://www.w3.org/TR/css-overflow-3/ - "none yet"
> [4] http://www.w3.org/TR/css3-conditional/ - separate links to shepherd
> and the harness (suite interface)
> [5] http://www.w3.org/TR/css-masking/ - links to the harness (runner)
> [6] http://www.w3.org/TR/filter-effects/ - links to the harness (runner)
> [7] http://test.csswg.org/suites/css3-transforms/nightly-unstable/


Rebecca,


>  1.  Add the Test Suite section to all of the specs – those with no
> tests labeled "none yet"

Just a thought here. Why not add a Test Suite section and a Take the
Test Harness section? Some will prefer the Test Suite, others taking
some tests in the Test Harness.

There is also a few other issues.

- A majority of tests have not been reviewed at all. There are tests
that are not precise and that are not correct out there. The spec maybe
in CR while its test suite status maybe less, much less mature, much
less reliable. There ought to be some indication of this prior to taking
tests.

- There are CSS2.1 tests that have comments, explanations with
corrections available and still have not been corrected yet.

- Some browsers, like Firefox (but not restricted to Firefox), require
to set some setting in order to try a particular test suite: eg.
layout.css.supports-rule.enabled must be enabled in order to try the CSS
Conditional Rules Test suite. Otherwise, the test suite for such browser
is useless.


If Test Suites and tests become more visible and widely available, it
may - in fact, I'm sure it will - easily lead to false claims, careless
exaggerations, premature claims about one browser over another.

Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 Test suite RC6, March 23rd 2011:
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

CSS 2.1 test suite harness:
http://test.csswg.org/harness/

Contributing to to CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html

Received on Tuesday, 25 June 2013 17:23:46 UTC