Re: Test Suite Language

Indeed, it would take some experimentation and research to figure out how
well this would work. Even if it doesn't work for _every_ test, we think it
would work for most tests. Since we need to have JS tests for JS features
(objects/enums/bindings, promises), there will always be a fallback if it's
impossible to use the tracing framework for some test.

On Fri, Sep 28, 2018 at 3:44 PM Filip Pizlo <fpizlo@apple.com> wrote:

> On Sep 28, 2018, at 3:28 PM, Kai Ninomiya <kainino@google.com> wrote:
>
> At the WebGL meeting we discussed a bit about this. The current state of
> the art of debugging C++ WASM in a browser is just not good enough for
> writing a test suite - we need quick development and good debugging (with
> ability to inspect variables/objects etc) in order to be able to co-develop
> the test suite along with the browser implementation.
>
> I believe asm.js+source maps would be somewhat more practical (although I
> remember frequent browser hangs when trying to debug such the huge generate
> JS file when compiling the dEQP OpenGL ES test suites).
>
>
> I see.  I think that asm.js is an OK solution if it gives a good debugging
> experience.  I think it makes sense to use that instead of wasm if it’s
> better than the wasm experience.
>
>
> The idea Myles is referring to of "producing Javascript code" is one we
> proposed in the meeting: Run a C++ program which can generate traces of
> WebGPU calls/expectations, and generate JS from the resulting traces. This
> would put some limitations on the form of the tests, but would output a
> hand-editable, easily-debuggable JS test file.
>
>
> Sounds like a neat idea.  I guess this is OK if it really works for all of
> the tests that you’re planning to write.
>
> -Filip
>
>
>
> On Fri, Sep 28, 2018 at 3:22 PM Filip Pizlo <fpizlo@apple.com> wrote:
>
>> I think that good tests have a transparent relationship between the
>> documented intent of the test to what the test is doing.  That suggests
>> minimizing the number of translation steps, but it’s not a showstopper if
>> there are translation steps.
>>
>> If the tests start out as C++, why not compile to WebAssembly?  Is
>> compiling to JavaScript better in some way?
>>
>> -Filip
>>
>>
>> > On Sep 28, 2018, at 2:21 PM, Myles C. Maxfield <mmaxfield@apple.com>
>> wrote:
>> >
>> > Hello, all
>> >
>> > Yesterday, we spent a while discussing which language the WebGPU test
>> suite should be written in. I thought I’d clarify some of our thoughts, as
>> I’m not sure they came through clearly in the discussion.
>> >
>> > Having a compilation step for our test suite is completely acceptable.
>> Nowadays, even documentation has compile steps, so it seems completely
>> reasonable for us to have one here as well. A compile step that compiles a
>> C++ program and runs it to produce Javascript code, is still just a compile
>> step. This is fine even if the structure of the produced Javascript isn’t
>> perfectly faithful to the original source code (aka a “trace”).
>> >
>> > We do have some desires for this test suite, though. First, we hope
>> that the test suite can be hosted on a web server, so users can run it with
>> a single tap/click (we feel this way about the WebGL test suite too).
>> Secondly, if there is a compilation step, it shouldn’t require an
>> incredibly complicated machine setup in order to build. Lastly, and perhaps
>> most importantly, the debugging experience for the Javascript tests
>> shouldn’t be too arduous. At the very least, we’d like to step-over
>> individual calls using Web Inspector, rather than having the test
>> represented in a JSON structure, for example. If we’re going to pursue the
>> compilation-that-produces-javascript strategy, we’d like to take a look at
>> a few examples of the produced Javascript before ramping up test
>> production, to see how difficult it would be to debug. There were also some
>> concerns yesterday that a trace may not be able to accurately capture the
>> intent of most tests, so that claim either needs to be demonstrated as
>> false, or some other Javascript structure should be produced.
>> >
>> > Given these thoughts, I hope we can reach a compromise.
>> >
>> > Thanks,
>> > Myles and Dean
>>
>>
>>
>>
>