Re: Test Suite Language

> On Sep 28, 2018, at 3:28 PM, Kai Ninomiya <kainino@google.com> wrote:
> 
> At the WebGL meeting we discussed a bit about this. The current state of the art of debugging C++ WASM in a browser is just not good enough for writing a test suite - we need quick development and good debugging (with ability to inspect variables/objects etc) in order to be able to co-develop the test suite along with the browser implementation.
> 
> I believe asm.js+source maps would be somewhat more practical (although I remember frequent browser hangs when trying to debug such the huge generate JS file when compiling the dEQP OpenGL ES test suites).

I see.  I think that asm.js is an OK solution if it gives a good debugging experience.  I think it makes sense to use that instead of wasm if it’s better than the wasm experience.

> 
> The idea Myles is referring to of "producing Javascript code" is one we proposed in the meeting: Run a C++ program which can generate traces of WebGPU calls/expectations, and generate JS from the resulting traces. This would put some limitations on the form of the tests, but would output a hand-editable, easily-debuggable JS test file.

Sounds like a neat idea.  I guess this is OK if it really works for all of the tests that you’re planning to write.

-Filip


> 
> On Fri, Sep 28, 2018 at 3:22 PM Filip Pizlo <fpizlo@apple.com <mailto:fpizlo@apple.com>> wrote:
> I think that good tests have a transparent relationship between the documented intent of the test to what the test is doing.  That suggests minimizing the number of translation steps, but it’s not a showstopper if there are translation steps.
> 
> If the tests start out as C++, why not compile to WebAssembly?  Is compiling to JavaScript better in some way?
> 
> -Filip
> 
> 
> > On Sep 28, 2018, at 2:21 PM, Myles C. Maxfield <mmaxfield@apple.com <mailto:mmaxfield@apple.com>> wrote:
> > 
> > Hello, all
> > 
> > Yesterday, we spent a while discussing which language the WebGPU test suite should be written in. I thought I’d clarify some of our thoughts, as I’m not sure they came through clearly in the discussion.
> > 
> > Having a compilation step for our test suite is completely acceptable. Nowadays, even documentation has compile steps, so it seems completely reasonable for us to have one here as well. A compile step that compiles a C++ program and runs it to produce Javascript code, is still just a compile step. This is fine even if the structure of the produced Javascript isn’t perfectly faithful to the original source code (aka a “trace”).
> > 
> > We do have some desires for this test suite, though. First, we hope that the test suite can be hosted on a web server, so users can run it with a single tap/click (we feel this way about the WebGL test suite too). Secondly, if there is a compilation step, it shouldn’t require an incredibly complicated machine setup in order to build. Lastly, and perhaps most importantly, the debugging experience for the Javascript tests shouldn’t be too arduous. At the very least, we’d like to step-over individual calls using Web Inspector, rather than having the test represented in a JSON structure, for example. If we’re going to pursue the compilation-that-produces-javascript strategy, we’d like to take a look at a few examples of the produced Javascript before ramping up test production, to see how difficult it would be to debug. There were also some concerns yesterday that a trace may not be able to accurately capture the intent of most tests, so that claim either needs to be demonstrated as false, or some other Javascript structure should be produced.
> > 
> > Given these thoughts, I hope we can reach a compromise.
> > 
> > Thanks,
> > Myles and Dean
> 
> 
>