W3C home > Mailing lists > Public > public-audio@w3.org > July to September 2013

Re: Reference files in the test suite

From: Ehsan Akhgari <ehsan.akhgari@gmail.com>
Date: Thu, 18 Jul 2013 14:15:28 -0400
Message-ID: <CANTur_7nEjM1qifBg5BxomU=be0f4x3_WBuvb5mq4exE=r6v5A@mail.gmail.com>
To: Chris Lowis <Chris.Lowis@bbc.co.uk>
Cc: "public-audio@w3.org" <public-audio@w3.org>
We should always strive for the first option.  This what I've been doing in
almost all Gecko test cases that I've written, and it is a lot more
reliable since the conditions for passing the test can all be expressed in
the code.  I would be very hesitant to use wave files produced by any of
the existing applications, because if the test fails on a different
implementation, it would be extremely difficult to figure out which
implementation is the buggy one.

We can explore option 2 if we ever want to test something for which we
cannot write code to generate the expected buffer, however I have no such
examples in mind right now.



On Thu, Jul 18, 2013 at 3:29 AM, Chris Lowis <Chris.Lowis@bbc.co.uk> wrote:

> Hello,
> The test of the GainNode in our test suite currently looks like this:
> https://github.com/w3c/web-platform-tests/blob/master/webaudio/the-audio-api/the-gainnode-interface/gain-node-test.html
> It's based on the layout test for the same node in the Webkit[1]. In it we
> compare a generate buffer to a known good reference, stored as a wav file.
> At the moment, I've lifted the reference files from the webkit layout
> tests, but, from the point of view of the W3C tests, it isn't particularly
> transparent as to where the reference comes from. There's a few options
> - generate a reference using mathematical operations in the setup of the
> test in an ArrayBuffer
> - generate a reference using a third-party software as a wave file
> (perhaps using something like Sox[2]), and have a script to document how it
> is produced
> - keep the wav files from the webkit source and use that as our reference.
> I think my preference is for the first option, but I wonder if someone
> familiar with the layout tests in blink/webkit (Chris?) could comment? I'd
> guess the references in those tests are there to catch regressions, and are
> regenerated using the engine itself periodically?
> Cheers,
> Chris
> [1]
> https://github.com/WebKit/webkit/blob/master/LayoutTests/webaudio/gain.html
> [2] http://sox.sourceforge.net/
> -----------------------------
> http://www.bbc.co.uk
> This e-mail (and any attachments) is confidential and
> may contain personal views which are not the views of the BBC unless
> specifically stated.
> If you have received it in
> error, please delete it from your system.
> Do not use, copy or disclose the
> information in any way nor act in reliance on it and notify the sender
> immediately.
> Please note that the BBC monitors e-mails
> sent or received.
> Further communication will signify your consent to
> this.
> -----------------------------
Received on Thursday, 18 July 2013 18:16:36 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:03:23 UTC