W3C home > Mailing lists > Public > public-css-testsuite@w3.org > June 2013

Re: Linking test suites to specs

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Tue, 25 Jun 2013 21:10:47 -0400
Message-ID: <eed11abd3c72b38cb618298e62f0b315.squirrel@ed-sh-cp3.entirelydigital.com>
To: "Rebecca Hauck" <rhauck@adobe.com>
Cc: "www-style list" <www-style@w3.org>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>

Le Mar 25 juin 2013 20:23, Rebecca Hauck a écrit :
> Hi Gérard,
> Thanks for your input on this! Comments inline below...
> On 6/25/13 10:23 AM, ""Gérard Talbot"" <css21testsuite@gtalbot.org>
> wrote:
>>Le Lun 24 juin 2013 19:30, Rebecca Hauck a écrit :
>>> Hi all,
>>> Some of the CSS specs have a "Test Suite" in the header, which I
>>> presume
>>> was part of an original spec template. However, in the specs I've
>>> come
>>> across that have it, it's either not used/updated [1][2][3], or it's
>>> used inconsistently [4][5][6].
>>> I was talking to fantasai about this a while ago (in the context of
>>> defining Test Owners at the spec level) and she suggested we link to
>>> the
>>> test suite cover page that is part of the nightly build. For example,
>>> [7].
>>> In an effort to make this consistent across all the the CSS specs and
>>> for the clearest test suite visibility (even so we can clearly see
>>> when
>>> there are none), I'm proposing the following:
>>>  1.  Add the Test Suite section to all of the specs ­ those with no
>>> tests labeled "none yet"
>>>  2.  Link to the test suite cover page per fantasai's suggestion
>>> If this is put into place, I'll then be modifying & using the test
>>> suite
>>> cover page to include things like test owner(s), coverage data, tests
>>> needed, etc.
>>> Any objections?
>>> May I add this to the agenda for this week's meeting? Once I have
>>> consensus, I'll make the updates accordingly.
>>> Thanks,
>>> -Rebecca
>>> [1] http://www.w3.org/TR/css3-transforms/ - "none yet"
>>> [2] http://www.w3.org/TR/css3-transitions/ - "none yet"
>>> [3] http://www.w3.org/TR/css-overflow-3/ - "none yet"
>>> [4] http://www.w3.org/TR/css3-conditional/ - separate links to
>>> shepherd
>>> and the harness (suite interface)
>>> [5] http://www.w3.org/TR/css-masking/ - links to the harness (runner)
>>> [6] http://www.w3.org/TR/filter-effects/ - links to the harness
>>> (runner)
>>> [7] http://test.csswg.org/suites/css3-transforms/nightly-unstable/
>>>  1.  Add the Test Suite section to all of the specs ­ those with no
>>> tests labeled "none yet"
>>Just a thought here. Why not add a Test Suite section and a Take the
>>Test Harness section? Some will prefer the Test Suite, others taking
>>some tests in the Test Harness.
> As Peter mentioned in a subsequent response to yours, the harness is
> already linked in the test annotation widgets (albeit on only the
> editor's
> drafts). He and I have added those annotations to a number of specs
> recently and will catch up with the rest missing them soon. (There
> shouldn't be that many more left at this point).  Keep in mind that I do
> intend to redesign that test suite landing page to include many more
> useful things than it currently has, so the harness link will definitely
> be there.

"to include many more useful things": great, then! Wonderful :)

> Of course, I'll solicit feedback from you and the rest of the
> group when I modify that page template.
> Also, the main point of this thread was to simply establish a clear and
> consistent link between each spec and its test suite. I personally don't
> have any objection to adding two links to the spec header instead of
> one,
> but unnecessary or redundant clutter may be concerns that other
> stakeholders have.
>>There is also a few other issues.
>>- A majority of tests have not been reviewed at all. There are tests
>>that are not precise and that are not correct out there. The spec maybe
>>in CR while its test suite status maybe less, much less mature, much
>>less reliable. There ought to be some indication of this prior to
>> taking
> Excellent point. I can summarize this on the test suite landing page.
> I'll
> work with Peter to see if I'd be able leverage a the Shepherd API here.
>>- There are CSS2.1 tests that have comments, explanations with
>>corrections available and still have not been corrected yet.
> This should also be addressed if I can leverage the Shepherd API as I
> presume all such tests are flagged there.
>>- Some browsers, like Firefox (but not restricted to Firefox), require
>>to set some setting in order to try a particular test suite: eg.
>>layout.css.supports-rule.enabled must be enabled in order to try the
>> CSS
>>Conditional Rules Test suite. Otherwise, the test suite for such
>> browser
>>is useless.
> I'm not sure how to respond to this in the context of the test suite
> page
> being linked to the specs. Hows is this requirement communicated now in
> Shepherd or the harness? And is a vendor-specific requirement really
> something that should be communicated by the W3C anyway? I'm inclined to
> think that the onus would be on the vendors actually running the tests
> to
> know what they're requirements are, but I'm open to your ideas on how
> this
> should be addressed here.

Yes, the vendors actually running the tests ought to know this... but
this may not be the case for anyone trying the tests. I hope Mozilla
would cancel such layout.css.supports-rule.enabled setting.

I vaguely recall - but I could be wrong here - that Chrome does that
also for some spec.

>>If Test Suites and tests become more visible and widely available, it
>>may - in fact, I'm sure it will - easily lead to false claims, careless
>>exaggerations, premature claims about one browser over another.
> Again, I'm not sure what you're suggesting here.
> Are you saying that the
> suites should _not_ be made more visible? Or just that we need to be
> careful to accurately describe what's in the test suite re: approved/not
> approved?  If the former, I very much disagree. In fact, I'd argue that
> the lower visibility a test suite has to the broader public, the more
> discrepant claims arise as typically (and historically) vendors develop
> their own test suites, which inevitably differ from the others. Perhaps
> I'm being simplistic, but the more visible a test suite is, the more
> implementors use it, which is sort of the point of having it in the
> first
> place.
> Further, it's very easy to find any given W3C spec, but often not as
> simple to find its tests -- it usually takes a couple more searches and
> a
> few more clicks, if you're savvy. Back to the simple point of this
> thread,
> the test suites should be consumed along with the specs and we should
> leverage the high discoverability of the specs to make the test suite
> just
> as accessible.  I definitely agree with you though that we need to be
> clear on the state of test suites.
> -Rebecca

Then, that's fine with me: more clarity about the state of test
suites... Whether tests have been reviewed and approved or if tests have
not reviewed yet and have not been approved yet. And the state of the
spec should be clear too: if a spec is a WD or a CR or a PR...

Contributions to the CSS 2.1 test suite:

CSS 2.1 Test suite RC6, March 23rd 2011:

CSS 2.1 test suite harness:

Contributing to to CSS 2.1 test suite:
Received on Wednesday, 26 June 2013 01:11:25 UTC

This archive was generated by hypermail 2.4.0 : Friday, 20 January 2023 19:58:19 UTC