Re: Calling All Tests

As mentioned by Paul at the F2F there is also a test suite in progress here

https://github.com/mohayonao/web-audio-test-api/


Matt

--
Matthew Paradis
Senior Software Engineer (Audio),
BBC Research & Development
030304 09889 | matthew.paradis@bbc.co.uk | www.bbc.co.uk/rd/sound

From: Joe Berkovitz <joe@noteflight.com<mailto:joe@noteflight.com>>
Date: Monday, 8 June 2015 22:06
To: Raymond Toy <rtoy@google.com<mailto:rtoy@google.com>>
Cc: Chris Lowis <chris.lowis@gmail.com<mailto:chris.lowis@gmail.com>>, Paul Adenot <padenot@mozilla.com<mailto:padenot@mozilla.com>>, Audio Working Group <public-audio@w3.org<mailto:public-audio@w3.org>>
Subject: Re: Calling All Tests
Resent-From: <public-audio@w3.org<mailto:public-audio@w3.org>>
Resent-Date: Monday, 8 June 2015 22:06


However, I was more concerned about the reverse.  We have the WPT repo, so how do I get that into Chrome's testing infrastructure without modifying any test in the WPT repo? Without this, blink won't find any utility in the repo and will probably just continue to use it's existing test suite.  Sadly.

I think that's an internal problem for browser vendors -- but assuming there is some suitably easy way to invoke a WPT suite and extract a result from it, then it shouldnt be hard.

If for some reason the WPT cannot be run that way, then I will be ready to argue that we should use some framework that is easy for vendors to include in their standard CI testing.


I assume that despite the failures you can still call yourself an implementation of WebAudio that purports to conform.

I guess that's how the world works, unfortunately. But if there is an agreed suite, it will at least get harder, and app developers are more likely to call out implementations on these points: they can run the tests too.


My assumption here was that, say, FF found a bug in their implementation and want a test for that and pushes that to the repo. Which then causes Chrome to fail that test (for whatever reason).  We wouldn't want to block FF from having a regression test, but we wouldn't want to be blocked from updating because the test will cause failures in our test bots.

Perhaps we need a way to gate tests in the suite so that you are able to arbitrarily exclude ones that are not applicable for some reason (or arbitrarily include the ones you want).

I won't have time to do this myself in the near future, but I would like to pull a few tests over from the WPT repo and run them to see how they would work in Chrome.   Perhaps everything will work just fine.

Thanks -- It'd be great to understand this landscape better.

...Joe



----------------------------

http://www.bbc.co.uk

This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------