W3C home > Mailing lists > Public > public-css-testsuite@w3.org > October 2015

Re: Simplifying metadata

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Thu, 29 Oct 2015 15:02:39 -0400
To: Geoffrey Sneddon <me@gsnedders.com>
Cc: Public CSS Test suite mailing list <public-css-testsuite@w3.org>
Message-ID: <3cd3b9384a8c2eb61eeb34097ca54b03@gtalbot.org>
Le 2015-10-29 02:05, Geoffrey Sneddon a écrit :
> On Thu, Oct 29, 2015 at 6:09 AM, Gérard Talbot 
> <css21testsuite@gtalbot.org>
> wrote:
> 
>> Le 2015-10-27 03:31, Geoffrey Sneddon a écrit :

[snipped]

>> I'm all for a discussion on metadata and documentation ... but what 
>> about
>> bad or wrong tests?
>> I wish
>> a) incorrect tests,
>> b) imprecise tests,
>> c) tests that can not fail (unreliable tests, non-trustworthy tests) 
>> and
>> d) tests that do not check what they claim to be checking
>> would be removed or rehabilitated or dealt with to start with. I've 
>> been
>> asking for this in the last 4 years (june 28th 2011 and even before 
>> that)
>> and it still has not been dealt with. And I'm not talking about a few 
>> dozen
>> tests here...
> 
> 
> a) and b) are quite easily found if browsers are actually running them

quite easily found? I have doubts...

> and
> are able to contribute fixes back upstream (which is another problem 
> we've
> had for years).

[snipped]

For CSS2.1 tests:

a) 65 CSS2.1 tests with Whiteboard NeedsWork=Incorrect
http://test.csswg.org/shepherd/search/testcase/spec/css21/status/issue/whiteboard/Incorrect/

b) 74 CSS2.1 tests with Whiteboard NeedsWork=Precision
http://test.csswg.org/shepherd/search/testcase/spec/css21/status/issue/whiteboard/Precision/


>> Personally, I think about 30% to 40% of all existing tests could be
>> re-engineered so that they would be associated with already available,
>> already created and very frequently reused reference files. When I 
>> create a
>> test, I always try to do this myself. That way,
>> a) I no longer have to think about creating a reference file,
>> b) this reduces server load when "doing" a test suite with the test
>> harness and
>> c) this reduces the growth of N reference files to be referenced
>> 
>> Examples given:
>> 
>> ref-if-there-is-no-red  : referenced by 290 tests (2 changesets!)
>> 
>> http://test.csswg.org/shepherd/search/reference/name/ref-if-there-is-no-red/
>> http://test.csswg.org/source/css21/reference/ref-if-there-is-no-red.xht
>> 
>> ref-this-text-should-be-green : referenced by 43 tests
>> test.csswg.org/shepherd/reference/ref-this-text-should-be-green/
>> 
>> http://test.csswg.org/source/css21/reference/ref-this-text-should-be-green.xht
>> 
>> So, why re-create 2 reference files that already exist?
> 
> 
> Because if there's 300 tests with "there should be no red below" and 
> 200
> with "there should be no red", it can easily be quite hard to remove 
> the
> word "below", because removing the word can cause later content to 
> reflow
> and the test to then fail.

[snipped]

Geoffrey, I tried to understand what you're saying and just could not. 
[Addendum: after more thinking, now I remember 1 test where what you 
described could *maybe* happen.]

Eg.

37 tests in
http://test.csswg.org/source/css-conditional-3/
are using, associating with the reference file
http://test.csswg.org/source/css-conditional-3/at-supports-001-ref.html
when it would be *_very easy_* to adapt those 37 tests to use, to link 
to
http://test.csswg.org/source/css21/reference/ref-filled-green-100px-square.xht

I probably could do this *_in less than_* 10 min. thanks to advanced 
search and replace.

The thing is: the current (and past) documentation are not encouraging 
test authors to reuse already created and available reference files.

Gérard
-- 
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html
Received on Thursday, 29 October 2015 19:03:16 UTC

This archive was generated by hypermail 2.3.1 : Thursday, 29 October 2015 19:03:19 UTC