Re: Getting browser vendors running and submitting tests

On 29/10/15 09:25, Gérard Talbot wrote:
> Le 2015-10-28 04:21, Geoffrey Sneddon a écrit :
>> [CC'd to public-test-infra; this is really mostly about CSS, can we avoid
>> cross-posting replies and keeping them on public-css-testsuite?]
>>
>> Myself and fantasai chaired a session at TPAC earlier today on this
>> subject. The minutes are at <
>> http://www.w3.org/2015/10/28-testing-minutes.html> (thanks fantasai!).
>>
>> Notably:
>>
>> * The build system makes it difficult for vendors to upstream their fixes
>> and their own tests (as vendors can't just fix the files they run; they
>> have to find some other file, fix that, and then work out how to build
>> the
>> test suite). As a result, they just fix the built files and don't
>> upstream
>> the fixes and often avoid updating the copy of the test suite because it
>> would trample their fixes.
>
> Why they don't fix their submitted tests (their tests in
> http://test.csswg.org/source/ ) and then re-import the modified test
> suite in their house system ?

They can, in theory, but won't, in practice. Typically developers have a 
choice between writing tests that will be upstreamed and writing tests 
that run in some single-browser system. They also have lots of pressure 
to land new features to meet goals, keep their browser competitive, and 
so on. Therefore they are likely to pick whatever testing system 
produces the least friction. A system where you have to edit files in 
one format then repull from upstream has way more friction than a system 
where you just edit the file that you want to change and commit it to 
your repository. Single browser test setups always have the latter 
workflow. We need to replicate that for cross-browser tests if we want 
developers to actually use them. With web-platform-tests we have managed 
this with gecko and servo, and are starting to see much more developer 
engagement with the web-platform-tests as a result.

Received on Thursday, 29 October 2015 01:25:39 UTC