Re: Reference files in the test suite

On Thu, Jul 18, 2013 at 12:29 AM, Chris Lowis <Chris.Lowis@bbc.co.uk> wrote:

> Hello,
>
> The test of the GainNode in our test suite currently looks like this:
>
>
> https://github.com/w3c/web-platform-tests/blob/master/webaudio/the-audio-api/the-gainnode-interface/gain-node-test.html
>
> It's based on the layout test for the same node in the Webkit[1]. In it we
> compare a generate buffer to a known good reference, stored as a wav file.
>
> At the moment, I've lifted the reference files from the webkit layout
> tests, but, from the point of view of the W3C tests, it isn't particularly
> transparent as to where the reference comes from. There's a few options
>
> - generate a reference using mathematical operations in the setup of the
> test in an ArrayBuffer
> - generate a reference using a third-party software as a wave file
> (perhaps using something like Sox[2]), and have a script to document how it
> is produced
> - keep the wav files from the webkit source and use that as our reference.
>
> I think my preference is for the first option, but I wonder if someone
> familiar with the layout tests in blink/webkit (Chris?) could comment? I'd
> guess the references in those tests are there to catch regressions, and are
> regenerated using the engine itself periodically?
>

I also generally prefer the first option when reasonably possible.  For
this particular test, it seems possible to compute the "true" result
mathematically.

Option 2 seems a bit troublesome since you now have to verify that the
third-party software also doesn't change the expected output in some way.
 Now we have two problems.

In general, references are regenerated when needed, and not periodically.
 When a test fails, I usually try to determine if the failure is expected
due to some changes in the code and that the actual output still makes
sense.  In that case, the reference is updated. Otherwise, there is
probably a true regression and that needs to be investigated, of course.



>
> Cheers,
>
> Chris
>
>
> [1]
> https://github.com/WebKit/webkit/blob/master/LayoutTests/webaudio/gain.html
> [2] http://sox.sourceforge.net/
>
>
> -----------------------------
> http://www.bbc.co.uk
> This e-mail (and any attachments) is confidential and
> may contain personal views which are not the views of the BBC unless
> specifically stated.
> If you have received it in
> error, please delete it from your system.
> Do not use, copy or disclose the
> information in any way nor act in reliance on it and notify the sender
> immediately.
> Please note that the BBC monitors e-mails
> sent or received.
> Further communication will signify your consent to
> this.
> -----------------------------
>
>