Re: Calling All Tests

First, most of Chrome's tests are located at
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/webaudio/

Almost all can be run in a browser without any special setup (using just
plain file:// access or a local http server).  There are a few manual tests
that require manual (duh!) interaction and interpretation.  Finally, there
are a few more tests for CORS access.

Second, I'd love to see all of the various test suites consolidated into
one test suite that we can share.

However, I do have some questions about this.

   - To be truly useful for Chrome, we would need to be able to import the
   testsuite directly and use it without having to modify anything. (Perhaps
   there is some setup code, but not the tests themselves.) This means, at
   least, that it can be run as is within Chrome's testing infrastructure so
   that we get continuous integration testing.
   - What exactly is the scope of this testsuite? Is it meant to be a kind
   of compliance test where if you pass, you are compliant?  And what if you
   don't pass? Does that mean you can't say you support WebAudio?
   - How accurate must the tests be?  There can be a lot of floating point
   going on, so differences between platforms and browsers are expected.  How
   is it decided that a result is accurate enough? We have this issue on
   Chrome today between Linux, OSX, Windows, and Android. This gets greatly
   multiplied once other browsers are added.
   - How will the tests be managed? Say a new test for Firefox is added but
   fails on Chrome. Now what? How is this resolved? Fisticuffs?
   - Who is responsible verifying the tests? And testing with the various
   browsers? Will this be a "reference" where each browser vendor can pull
   from and use?  And then any issues get raised to be resolved by the group?


Despite these questions, I wholeheartedly support a unified testsuite. I
just want to know how it will work in practice.


On Mon, Jun 8, 2015 at 8:58 AM, Chris Lowis <chris.lowis@gmail.com> wrote:

> Hi Joe,
>
> The W3C has a test suite which has some basic Web Audio tests in it:
> https://github.com/w3c/web-platform-tests/tree/master/webaudio.
>
> A while ago I wrote a contribution guide, including how to import
> tests from the various vendors' own test suites which your intern
> might find helpful:
> http://blog.chrislowis.co.uk/2014/04/30/testing-web-audio.html
>
> I am able to review/approve pull requests to the web platform tests
> test suite, and off your intern some guidance as to format/style etc.
> if helpful.
>
> Best,
>
> Chris
>
>
>
>
> On 8 June 2015 at 16:34, Joe Berkovitz <joe@noteflight.com> wrote:
> > Hello Paul, Raymond,
> >
> > I am following up on our decision at the F2F to try to create a unified
> JS
> > test suite for Web Audio that will bring together existing tests
> contributed
> > by the browser vendors, and also expand to include other tests
> contributed
> > by WG members and the developer community at large.
> >
> > Noteflight has an intern available that is able to assist with this task,
> > starting now. I would very much like to start this person off on
> coalescing
> > the existing tests from implementations into a single suite that is
> runnable
> > independent of browser-specific tooling. This can then be checked into a
> W3C
> > repo alongside the spec.
> >
> > Of course, we're not seeking to preempt anyone else who wants to do this
> > instead :-) I just want to get the ball rolling, and we're ready to help.
> >
> > What is the best way to get started on this? Can someone from your
> > respective teams provide a few representative test samples from Chromium
> and
> > Mozilla to get going, and then a larger dump later? Or is there a better
> way
> > to proceed?
> >
> > Best,
> >
> > .            .       .    .  . ...Joe
> >
> > Joe Berkovitz
> > President
> >
> > Noteflight LLC
> > 49R Day Street / Somerville, MA 02144 / USA
> > phone: +1 978 314 6271
> > www.noteflight.com
> > "Your music, everywhere"
>