Re: Test Suite Language

At the WebGL meeting we discussed a bit about this. The current state of
the art of debugging C++ WASM in a browser is just not good enough for
writing a test suite - we need quick development and good debugging (with
ability to inspect variables/objects etc) in order to be able to co-develop
the test suite along with the browser implementation.

I believe asm.js+source maps would be somewhat more practical (although I
remember frequent browser hangs when trying to debug such the huge generate
JS file when compiling the dEQP OpenGL ES test suites).

The idea Myles is referring to of "producing Javascript code" is one we
proposed in the meeting: Run a C++ program which can generate traces of
WebGPU calls/expectations, and generate JS from the resulting traces. This
would put some limitations on the form of the tests, but would output a
hand-editable, easily-debuggable JS test file.

On Fri, Sep 28, 2018 at 3:22 PM Filip Pizlo <fpizlo@apple.com> wrote:

> I think that good tests have a transparent relationship between the
> documented intent of the test to what the test is doing.  That suggests
> minimizing the number of translation steps, but it’s not a showstopper if
> there are translation steps.
>
> If the tests start out as C++, why not compile to WebAssembly?  Is
> compiling to JavaScript better in some way?
>
> -Filip
>
>
> > On Sep 28, 2018, at 2:21 PM, Myles C. Maxfield <mmaxfield@apple.com>
> wrote:
> >
> > Hello, all
> >
> > Yesterday, we spent a while discussing which language the WebGPU test
> suite should be written in. I thought I’d clarify some of our thoughts, as
> I’m not sure they came through clearly in the discussion.
> >
> > Having a compilation step for our test suite is completely acceptable.
> Nowadays, even documentation has compile steps, so it seems completely
> reasonable for us to have one here as well. A compile step that compiles a
> C++ program and runs it to produce Javascript code, is still just a compile
> step. This is fine even if the structure of the produced Javascript isn’t
> perfectly faithful to the original source code (aka a “trace”).
> >
> > We do have some desires for this test suite, though. First, we hope that
> the test suite can be hosted on a web server, so users can run it with a
> single tap/click (we feel this way about the WebGL test suite too).
> Secondly, if there is a compilation step, it shouldn’t require an
> incredibly complicated machine setup in order to build. Lastly, and perhaps
> most importantly, the debugging experience for the Javascript tests
> shouldn’t be too arduous. At the very least, we’d like to step-over
> individual calls using Web Inspector, rather than having the test
> represented in a JSON structure, for example. If we’re going to pursue the
> compilation-that-produces-javascript strategy, we’d like to take a look at
> a few examples of the produced Javascript before ramping up test
> production, to see how difficult it would be to debug. There were also some
> concerns yesterday that a trace may not be able to accurately capture the
> intent of most tests, so that claim either needs to be demonstrated as
> false, or some other Javascript structure should be produced.
> >
> > Given these thoughts, I hope we can reach a compromise.
> >
> > Thanks,
> > Myles and Dean
>
>
>
>