Re: Reference files in the test suite

On Thu, Jul 18, 2013 at 12:29 AM, Chris Lowis <Chris.Lowis@bbc.co.uk> wrote:

> Hello,
>
> The test of the GainNode in our test suite currently looks like this:
>
>
> https://github.com/w3c/web-platform-tests/blob/master/webaudio/the-audio-api/the-gainnode-interface/gain-node-test.html
>
> It's based on the layout test for the same node in the Webkit[1]. In it we
> compare a generate buffer to a known good reference, stored as a wav file.
>
> At the moment, I've lifted the reference files from the webkit layout
> tests, but, from the point of view of the W3C tests, it isn't particularly
> transparent as to where the reference comes from. There's a few options
>
> - generate a reference using mathematical operations in the setup of the
> test in an ArrayBuffer
> - generate a reference using a third-party software as a wave file
> (perhaps using something like Sox[2]), and have a script to document how it
> is produced
> - keep the wav files from the webkit source and use that as our reference.
>
> I think my preference is for the first option, but I wonder if someone
> familiar with the layout tests in blink/webkit (Chris?) could comment? I'd
> guess the references in those tests are there to catch regressions, and are
> regenerated using the engine itself periodically?
>

Hi Chris, I think we talked some about this at the face-face, but I agree
with Ehsan that the tests which compare against a reference .wav file are
not the best tests.  We have written a few of these in the WebKit/Blink
layout tests, and in some cases we could convert them to better tests
written to test against a reference generated purely in JavaScript, which
is how most of our layout tests work...

Cheers,
Chris



>
> Cheers,
>
> Chris
>
>
> [1]
> https://github.com/WebKit/webkit/blob/master/LayoutTests/webaudio/gain.html
> [2] http://sox.sourceforge.net/
>
>
> -----------------------------
> http://www.bbc.co.uk
> This e-mail (and any attachments) is confidential and
> may contain personal views which are not the views of the BBC unless
> specifically stated.
> If you have received it in
> error, please delete it from your system.
> Do not use, copy or disclose the
> information in any way nor act in reliance on it and notify the sender
> immediately.
> Please note that the BBC monitors e-mails
> sent or received.
> Further communication will signify your consent to
> this.
> -----------------------------
>
>