Re: Who currently executes the tests in the w3c repos?

On Aug 7, 2013, at 2:22 AM, Ms2ger wrote:

> On 08/07/2013 02:43 AM, Dirk Pranke wrote:
>> Hi all,
>> 
>> I am a relative newcomer to this group but I have been working off and on
>> recently (quite a bit just now) getting the tests running as part of the
>> automated tests for Blink and WebKit.
>> 
>> I believe I'm probably missing quite a bit of context or history that makes
>> it difficult for me to understand some of the design decisions and
>> processes around getting tests written and submitted and run.
>> 
>> So forgive me if this sounds like a brash question, but it's honestly one
>> coming from ignorance and not meant to be snarky ...:
>> 
>> Who actually currently runs these tests, and how?
>> 
>> As far as I know, no one in Blink (or WebKit) regularly runs any of these
>> tests, even manually, with a few exceptions where we have manually imported
>> some suites into our existing repos. It may also be the case that some
>> times individual developers or spec editors have run some of the tests.
>>> From my limited conversations w/ Fantasai, I believe the situation is
>> similar for Mozilla. I do not know about efforts inside Microsoft or at
>> Opera, or at any other browser vendor or third party.
>> 
>> Are there groups that actually do attempt to run the tests somehow on the
>> different browsers? Does that somehow happen in Shepherd in a way I don't
>> know about (or understand)?
>> 
>> I would like to be able to usefully contribute to threads like
>> "consolidating css-wg and web-platform-tests repositories" and talk about
>> the pain points I'm hitting as I try to get the tests running, but it's
>> hard for me to say useful things w/o knowing more about how others are
>> using all of this. So, I'm looking to become educated?
> 
> Mozilla imports a subset of the web-platform-tests repository into its main repository (based on the MANIFEST files you might have noticed). The importing is completely scripted, including writing the annotations for failing tests. The code for that is available [1]. Those tests are run both in automation and by developers locally from there.
> 
> I understood Ryosuke Niwa was working on importing tests in WebKit before the fork; have you talked to him?
> 
> There are various tools to run tests semi-automatically [2-4].
> 
> The Shepherd tool [5] is a Test Suite Manager used by the CSS WG for reviewing; there are some review comments there, but in my personal experience those are largely ignored. This may be because there is no value in getting tests reviewed; the CSS WG only cares about tests to get specifications to CR, and for that purpose, unreviewed (and probably incorrect) tests are used.

This statement is untrue and misleading. Firstly, our primary focus of testing is getting specs _out_ of CR and into REC, getting to CR does not require tests, but this is not the "only" thing we care about regarding tests. We do care about the health of the platform and want to promote proper, interoperable implementations as well. However, we currently have approximately 60 specifications on the REC track and we do respect the REC track process. A lack of testing is one of our primary blockers getting specs to REC so this is a major focus for us.

Secondly, this implies that we're gaming the system and don't care about the quality of the tests we use, which is completely false. We used to have a policy of not including tests in our suites until they were reviewed. We found that our review backlog was too long (by thousands of tests) and changed our review policy to allow unreviewed tests into the suites and allow the reviews to be triggered by looking for unexpected results. Yes, there are most likely a small percentage of tests that are incorrect, but the review backlog is still being worked even after the relevant spec has exited CR and the test suite shifted into the conformance stage.

Received on Wednesday, 7 August 2013 16:33:31 UTC