Re: Single file tests

On 02/04/14 10:57, Robin Berjon wrote:
> On 02/04/2014 10:30 , Simon Pieters wrote:
>> On Wed, 02 Apr 2014 00:07:49 +0200, James Graham
>> <james@hoppipolla.co.uk> wrote:
>>> So one request I've had from people at Mozilla used to writing
>>> Mochitests is an easier way to write testharness.js tests,
>>> particularly for the special case where there is exactly one test per
>>> file. This is a interesting case because the file itself provides the
>>> isolation that we usually try to obtain by wrapping each test step in
>>> a function() {}, so it is possible to cut out some of the verbosity.
>>> An example of the kind of result we might get is
>>>
>>> <doctype html>
>>> <title>Example test</title>
>>> <script src=/resources/testharness.js></script>
>>> <script src=/resources/testharnessreport.js></script>
>>> <script>
>>> setup({file_is_test:true})
>>>
>>> onload = function() {assert_true(true); done()}
>>>
>>> I think there are some possible disadvantages to this; in particular
>>> it might encourage people to put a whole load of things that should be
>>> separate tests as multiple asserts in one file in a way that will
>>> break if one assert doesn't work in a particular browser. However it
>>> does seem like if used wisely it could be a win, and might help sell
>>> the idea of writing testharness.js tests going forward at Mozilla.
>>> What do people think?
>>
>> I think it's a good idea. But it would be nice to minimize the
>> boilerplate.
>
> Agreed.
>
>> The snippet above omits <div id="log"></div> which is currently
>> required, but the script could create it if it's missing.

OK, I updated the branch[1] to not require <div id=log> (ever) and to 
assume that a bare assert_foo before any test has been created implies 
the file is a single test. This does mean that the detection for "has 
the test file finished" has to change so that a file with 0 tests will 
sit and look blank until it finally times out. Which I guess is OK, but 
could be a bit confusing.

> The script could also inject the link to testharnessreport.js if it
> isn't there? It's a little bit tricky due to existing content but maybe
> given a flag? The flag wouldn't be very useful if that's all it did, but
> it could also take care of setup({file_is_test:true}).
>
> Also, do we really need to run onload for most tests?
>
>    <!doctype html>
>    <title>Example test</title>
>    <script src=/resources/testharness.js data-test></script>
>    <script>
>    onload = function() {assert_true(true); done(); };
>
> Crudely:
>
> if (document.currentScript.dataset.test !== undefined) {
>      var s = document.createElement("script");
>      s.src = document.currentScript.src.replace("testharness",
> "testharnessreport");
>      s.onload = function () { setup({file_is_test:true}); };
>      document.currentScript.after(s);
> }

I worry that doing DOM magic before the tests have run leads to 
hard-to-debug issues. For example if someone has a test that does

document.getElementsByTagName("script")[3]

and they only count the scripts that they can see on the page, they will 
have issues. The more DOM manipulation that the harness does, the less 
likely it is to run in new UAs; using things like postMessage has 
already been a problem for Servo (and onerror may have similar problems, 
but maybe the benefits are worth it). So all in all I think that 
explicit is better than implicit here.

>    <!doctype html>
>    <title>Example test</title>
>    <script src=/resources/testharness.js></script>
>    <script>
>    run(function() {assert_true(true); done(); });

I don't want to special-case things like this. There are many tests that 
won't depend on onload, but might depend on some other events that 
should be fired.

[1] https://critic.hoppipolla.co.uk/r/1151

Received on Wednesday, 2 April 2014 10:38:56 UTC