Re: Who currently executes the tests in the w3c repos?

To augment what Ms2ger said, we (Mozilla) are going to be working on 
importing a lot more tests and getting those running in our 
infrastructure a lot more. The Spec I am working on[1] needs more tests 
in our tree. This is going to be worked on within the next quarter.

David

On 07/08/2013 10:22, Ms2ger wrote:
> On 08/07/2013 02:43 AM, Dirk Pranke wrote:
>> Hi all,
>>
>> I am a relative newcomer to this group but I have been working off 
>> and on
>> recently (quite a bit just now) getting the tests running as part of the
>> automated tests for Blink and WebKit.
>>
>> I believe I'm probably missing quite a bit of context or history that 
>> makes
>> it difficult for me to understand some of the design decisions and
>> processes around getting tests written and submitted and run.
>>
>> So forgive me if this sounds like a brash question, but it's honestly 
>> one
>> coming from ignorance and not meant to be snarky ...:
>>
>> Who actually currently runs these tests, and how?
>>
>> As far as I know, no one in Blink (or WebKit) regularly runs any of 
>> these
>> tests, even manually, with a few exceptions where we have manually 
>> imported
>> some suites into our existing repos. It may also be the case that some
>> times individual developers or spec editors have run some of the tests.
>>> From my limited conversations w/ Fantasai, I believe the situation is
>> similar for Mozilla. I do not know about efforts inside Microsoft or at
>> Opera, or at any other browser vendor or third party.
>>
>> Are there groups that actually do attempt to run the tests somehow on 
>> the
>> different browsers? Does that somehow happen in Shepherd in a way I 
>> don't
>> know about (or understand)?
>>
>> I would like to be able to usefully contribute to threads like
>> "consolidating css-wg and web-platform-tests repositories" and talk 
>> about
>> the pain points I'm hitting as I try to get the tests running, but it's
>> hard for me to say useful things w/o knowing more about how others are
>> using all of this. So, I'm looking to become educated?
>
> Mozilla imports a subset of the web-platform-tests repository into its 
> main repository (based on the MANIFEST files you might have noticed). 
> The importing is completely scripted, including writing the 
> annotations for failing tests. The code for that is available [1]. 
> Those tests are run both in automation and by developers locally from 
> there.
>
> I understood Ryosuke Niwa was working on importing tests in WebKit 
> before the fork; have you talked to him?
>
> There are various tools to run tests semi-automatically [2-4].
>
> The Shepherd tool [5] is a Test Suite Manager used by the CSS WG for 
> reviewing; there are some review comments there, but in my personal 
> experience those are largely ignored. This may be because there is no 
> value in getting tests reviewed; the CSS WG only cares about tests to 
> get specifications to CR, and for that purpose, unreviewed (and 
> probably incorrect) tests are used.
>
> The group working on the web-platform-tests repository seems to 
> primarily have another goal: improving interoperability of the web 
> platform (between web browsers in particular), with little interest in 
> the W3C process. The fact that most of those specifications are 
> primarily developed in the WHATWG and a lot of contributors work in 
> browser QA may be factors in that. To achieve that goal, we have 
> imposed a review-then-commit policy, which introduces a bigger 
> incentive for reviewers: if tests are not reviewed, they're not in the 
> main repository, and they're harder to run / import / ...
>
> I hope this helps to sketch the current playing field in web standards 
> testing; do not hesitate to get in touch if there's anything else I 
> can help with, on this mailing list, or on IRC (irc.w3.org/testing or 
> irc.freenode.net/whatwg).
>
> Ms2ger
>
> [1] http://mxr.mozilla.org/mozilla-central/source/dom/imptests/
> [2] https://bitbucket.org/ms2ger/test-runner
> [3] http://www.w3c-test.org/framework
> [4] https://test.csswg.org/harness/
> [5] http://test.csswg.org/shepherd/
>

Received on Wednesday, 7 August 2013 12:47:41 UTC