Re: Unifying testsuite policy and getting rid of CSS exceptions

On 14/09/17 23:41, Alan Stearns wrote:
> And keeping the spec link metadata in CSS tests is a requirement we’re going to maintain.

I think this is a very unfortunate requirement. It isn't required by any 
other spec, and it is yet another obstacle to people writing 
cross-browser tests instead of single vendor tests. Particularly for 
reftests, almost any test could be a shared test, but looking at the 
data most new gecko reftests are not being contributed upstream, and I 
guess that's the same for blink.

That should be regarded as a serious problem, because it has a clear 
negative impact on our ability to ship interoperable features and build 
a healthy, reliable platform.

As far as I can tell there are basically two common objections:

* CSSWG needs to do (process stuff) which means we need this data to get 
specs to Rec.

* Having those links is helpful to understand what a test is testing.

For the first, I note again that no other working group has this 
requirement, and yet many have managed to get specs to Rec. If CSS want 
data on which tests should form part of the implementation report they 
can and should maintain that externally rather than making test authors 
do the work with the predictable result that they simply don't bother 
and write tests without the additional requirements. I'm even happy for 
that metadata to be in the tests, as long as it isn't a requirement 
placed on test authors.

For the second, I agree obviously that knowing what's being tested is 
good. But I don't think this data is sufficiently useful compared to the 
cost of maintaining it that it should be required everywhere. In 
practice anyone sufficiently knowledgeable to be interacting with a test 
can likely identify the relevant parts of the specs rather quickly.

In general, I think it's worth considering the cost to the wider 
ecosystem of policies and requirements, because things that help one 
specific group can nevertheless be a net negative when the full picture 
is considered.

Received on Friday, 15 September 2017 10:55:24 UTC