Re: Calling All Tests

Thanks for all the great responses everyone. I'll try to respond to
Raymond's points as best I can just to advance the discussion, since Paul
seconded his questions. This is just a starting point...

However, I do have some questions about this.
>
>    - To be truly useful for Chrome, we would need to be able to import
>    the testsuite directly and use it without having to modify anything.
>    (Perhaps there is some setup code, but not the tests themselves.) This
>    means, at least, that it can be run as is within Chrome's testing
>    infrastructure so that we get continuous integration testing.
>
>
If we upstreamed everything from the Chrome and Moz suites into the WPT
repo (which already contains a few Web Audio tests), would that suffice?

>
>    - What exactly is the scope of this testsuite? Is it meant to be a
>    kind of compliance test where if you pass, you are compliant?  And what if
>    you don't pass? Does that mean you can't say you support WebAudio?
>
> My belief is that this should be a test suite that identifies
noncompliance in some set of areas. So if you fail the tests, your
implementation is definitely not compliant.

However I don't think we need to sign up for the converse being true (that
if you pass the tests, your implementation is compliant). That would
require the suite to achieve total coverage in the testing of every aspect
of the API including many obscure issues we have not thought of or defined
yet. So I believe the test suite will continue to be an evolving work in
progress, and thus can't absolutely assure compliance.


>    - How accurate must the tests be?  There can be a lot of floating
>    point going on, so differences between platforms and browsers are
>    expected.  How is it decided that a result is accurate enough? We have this
>    issue on Chrome today between Linux, OSX, Windows, and Android. This gets
>    greatly multiplied once other browsers are added.
>
> We should have some definable threshold for errors that we agree on.
Obviously exact floating point comparisons are not going to work. I think
proposals are needed.

To aid in this, any fuzzy comparison techniques should be defined in some
central place that applies across the test suite, so that the approach can
be adjusted as the group moves forward.


>
>    - How will the tests be managed? Say a new test for Firefox is added
>    but fails on Chrome. Now what? How is this resolved? Fisticuffs?
>
> How is this done today for specs that already possess extensive and
reasonably agreed test suites?

I would think if there are conflicts in test results which vendors don't
agree on, then they ultimately reflect problems with the spec that the
group needs to address. Today we seem to manage this kind of progress
mostly without fisticuffs (although it is tempting, I know...)

So the Chrome team could a) agree the test is valid and file a bug (and
eventually fix Chrome), b) claim that it's invalid and take a spec dispute
to the WG, which could go either way.

By the way, my hope would be that new tests are not added specifically "for
FF" or "for Chrome" but are added to a central repo from which all vendors
would periodically pull.


>    - Who is responsible verifying the tests? And testing with the various
>    browsers? Will this be a "reference" where each browser vendor can pull
>    from and use?  And then any issues get raised to be resolved by the group?
>
>
>
I am a little unsure on the best way to proceed for QA on the tests
themselves. I think once we pull them all together and can run them on all
the major platforms, we'll have a better idea how much stuff needs to be
reconciled and how complicated the situation on the ground actually is.

Ultimately, though, I believe we should wind up with a single reference
suite of tests that run successfully on all compliant browsers, and any new
tests will need to run through some sort of approval process to get into
that suite. Probably we need to have some conventions with respect to
release and dev branches of the test suite itself.

...Joe