Fwd: status of CSS Backgrounds and Borders Module Level 3 Tests

Hi,

After some work done on the CSS Backgrounds and Borders Module Level 3 
CR Test Suite (thanks to David Baron and Gérard Talbot ) and after 
running the test suite against multiple browsers, the results have 
currently largely improved.

In my previous message, there was only 372 of 603 required tests meeting 
CR exit criteria.
Today we have 479 of 604 required tests meet CR exit criteria.
- 2 of 2 optional tests meet CR exit criteria.
- 125 required tests are considered valid and do not meet CR exit criteria.

I couldn't run these tests on Safari, as I do not have a Apple computer.
Running those on Safari (especially the one highlighted in yellow) may 
improve the results.

The CSS Backgrounds and Borders Module Level 3 CR Test Suite Results
are available at
http://test.csswg.org/suites/css3-background/nightly-unstable/report/results.html


Another issue is the proper coverage of this testsuite.

Some sections have a large number of tests (250), some just one or two 
tests. Somme sections have no tests.  (Attached is a file with the 
number of test per sections)
So the question remains about the testsuite being good enough to cover 
all the functionalities of the spec.

Thierry.






-------- Message transféré --------
From: Thierry MICHEL <tmichel@w3.org>
Subject: status of CSS Backgrounds and Borders Module Level 3 Tests
References: <b5f4db10-41f9-0ec1-0078-183248498a75@w3.org>
To: w3c-css-wg@w3.org

Hi,

For sometime I have been running the CSS Backgrounds and Borders Module 
Level 3 tests against multiple implementations.

CSS Backgrounds and Borders Module Level 3 is a W3C Candidate 
Recommendation 9 September 2014
https://www.w3.org/TR/css3-background/

The CSS Backgrounds and Borders Module Level 3 CR Test Suite Results
are available at
http://test.csswg.org/suites/css3-background/nightly-unstable/report/results.html

Edge implementation which had not been earlier tested against the 
testsuite, now allowed to have a fair number of tests to have now at 
least two passing implmentations.
Bert also helped running these tests on Safari.

Currently 372 of 603 required tests meet CR exit criteria.

But a large number of tests (231) still fail passing 2 implementations 
(see list following)

Also a large set of tests that are unsuable, it seem impossible to know 
if they pass or not, because they have neither a description of the 
expected result nor a reference.

I don't know where these tests come from, but they need to be updated 
for usability.
Here are a few examples of these tests:

http://test.csswg.org/harness/test/css-backgrounds-3_dev/spec/css-backgrounds-3/section/3.9/tall--auto-32px--nonpercent-width-percent-height-viewbox/

http://test.csswg.org/harness/test/css-backgrounds-3_dev/spec/css-backgrounds-3/section/3.9/tall--auto-32px--omitted-width-nonpercent-height/


If you would like to run these tests you should use the CSS Backgrounds 
and Borders Module Level 3 CR Test Suite
http://test.csswg.org/harness/suite/CSS3-BACKGROUND_DEV/



List of tests failling to pass 2 implementations:
-------------------------------------------------

3.3: Image Sources: the ‘background-image’ property
7 tests not passing

3.4: Tiling Images: the ‘background-repeat’ property
2 tests not passing

3.7: Painting Area: the ‘background-clip’ property
12 tests not passing

3.8: Positioning Area: the ‘background-origin’ property
12 tests not passing

3.9: Sizing Images: the ‘background-size’ property
most of these tests not passing

3.10: Backgrounds Shorthand: the ‘background’ property
1 test not passing

4.1: Line Colors: the ‘border-color’ properties
1 test not passing

6.2: Image Slicing: the ‘border-image-slice’ property
2 tests not passing

6.5: Image Tiling: the ‘border-image-repeat’ property
2 tests not passing

7.1: Drop Shadows: the ‘box-shadow’ property
1 test not passing


Best,

Thierry