Re: HTML test results

On Wed, Jun 4, 2014 at 9:38 AM, Robin Berjon <robin@w3.org> wrote:

> Hi group,
>
> just a heads up that we have been running the test suite against
> implementations and have been gathering updated results for the HTML CR.
> They are not final yet, and will get updated, but the current status is
> documented here:
>
>     http://w3c.github.io/test-results/html/less-than-2.html
>     http://w3c.github.io/test-results/html/details.html


There are other factors that skew these results as well, and here I am
thinking of the aggregation of multiple asserts into a single 'test'. For
example, in [1], we see from the results that 5 of 6 tested browsers fail
all tests.

If one examines one of the tests, e.g., we see three asserts that are
potentially unrelated: (1) is property in window, (2) is property in
Window.prototype, (3) is property descriptor of property in window
!writable, enumerable, and configurable.

  readonlyAttributes.forEach(function(id) {
    test(function() {
      var WindowProto = Window.prototype;
      assert_true(id in window, id + " in window");
      assert_false(id in WindowProto, id + " in Window.prototype");
      assert_accessor_propdesc(Object.getOwnPropertyDescriptor(window, id),
                               false, true, true);
    }, "Window readonly attribute: " + id);
  });
Given the relative poor implementation status of WebIDL related semantics
on host objects, fails from the last two asserts are swamping passes from
the first assert. Since most content authors rely exclusively on the
semantics of the first assert, but very few rely on the WebIDL related
semantics, the organization of this test produces a worse picture of
interoperability than one would get from looking at only property presence
instead of adding in WebIDL semantics.

[1]
http://www.w3c-test.org/html/browsers/the-window-object/window-properties.html