W3C home > Mailing lists > Public > public-test-infra@w3.org > January to March 2011

[minutes] Call 2011-03-21

From: Francois Daoust <fd@w3.org>
Date: Tue, 22 Mar 2011 14:22:50 +0100
Message-ID: <4D88A2AA.6080500@w3.org>
To: public-test-infra@w3.org
Minutes of yesterday's call:

... copied as raw text below.

Discussion around:
- Shadi's updates to requirements page:
- Review open-source Web test suite:


21 Mar 2011

    See also: [2]IRC log

       [2] http://www.w3.org/2011/03/21-testing-irc


           Plh, Shadi, francois


           plh, francois


      * [3]Topics
          1. [4]List of requirements
          2. [5]Potential contributor
      * [6]Summary of Action Items

List of requirements

    Francois: still looking at the Mozilla test framework. they are many
    ways that are used to test firefox
    ... some are testing the chrome
    ... some are testing the display engine
    ... hard to differentiate
    ... so, still looking at my way in
    ... most of JS tests rely on having some extensions installed

    shadi: the distinction between the chrome and the browser is

    -> [7]https://developer.mozilla.org/en/Mozilla_automated_testing
    Mozilla automated testing (for starters)

       [7] https://developer.mozilla.org/en/Mozilla_automated_testing

    Shadi: for test execution section, it's still a bit too vague

    plh: I don't mind targeting a main context that is a Web browser
    with HTML/CSS/JavaScript support.

    shadi: there may different types of tests and different ways to
    approach these tests, to run them.

    plh: Yes, I agree. For screenshots, it may be JavaScript-based, or
    using some external tool, or manual.

    shadi: we should look at these classes of tests and precise them a
    little bit, to see if we need different methods to run them.

    plh: we've been playing with some of these for some time.
    ... e.g. comparing plain text output is not as easy to do
    cross-platform. So that's why we separated from JavaScript testable

    shadi: maybe we could link to examples from the Wiki in terms of
    testing method

    <scribe> ACTION: plh to add examples to the Wiki in terms of testing
    methods [recorded in

    shadi: back to the comment that main target has JavaScript enabled,
    I think we may have limitations when we're testing JavaScript.

    plh: we know already that there are tests that cannot be tested with
    ... I just do not want to spend ages trying to see how to test a
    browser that does not have JavaScript.
    ... Most CSS tests do not have JavaScript inside them. Many HTML
    tests do not have JavaScript either. A good example is parsing

    shadi: one of the issues is how assistive technologies play with the
    DOM and we are willing, as part of ARIA, to test JavaScript support
    for some aspects.

    plh: yes, relying on JavaScript for some of the tests is a non
    ... Back at your inputs, I see you added media players. What do you
    mean by that?

    shadi: I was not only thinking about HTML5, more in terms of generic
    captioning, e.g. video online.

    plh: well, that's HTML5, in my view.
    ... I can think of a good example that is not HTML: SVG video.
    ... We may not have HTML around, but we still have JavaScript
    ... as part of HTML testing, we're not going to test the properties
    of plug-ins, inside the object element.
    ... That's not our job.
    ... If you guys define some generic constraints on video playback
    because you want to be technology independent, that may be

    shadi: For instance, if a user tabs between elements in a Web page
    that has some video running, no matter the method, it often breaks
    the accessibility of the page.

    plh: I would think that it is part of the HTML testing, as it is to
    make sure that keyboard navigation behaves as expected, but the fact
    that it's a video doesn't really matter.
    ... in the end, it will probably fall into human tests, i.e. I can't
    think of a way to automate that kind of tests.

    shadi: what about SMIL?

    plh: not currently a priority.
    ... In terms of technologies, there is a subset of technologies that
    I have in mind that range across 15 or so working groups.
    ... I could be wrong, but what you're suggesting around
    accessibility APIs does not seem to be doable on an automated basis.
    ... manual tests we have in mind use a pass/fail button. It's easy.
    It's our fallback in terms of testing method.

    shadi: I did not see anything in the proposal that striked me as
    "non-accessibility friendly", so more working groups may align to

    <plh-home> [9]http://w3c-test.org/resources/testharness.js

       [9] http://w3c-test.org/resources/testharness.js

    francois: wonder if we should put the set of dependencies that one
    can use when one writes a test case, e.g. a Javascript test or a CSS

    plh: Interesting. For JavaScript, you can rely on the test harness I
    just pasted.
    ... It could be used in a higher-level framework.
    ... I actually started to write tests using that harness on Friday,
    because I realized there were a few troubles to be fixed before we
    ask people to use it.
    ... For human tests, that's easy, you just explain the test to the
    ... For screenshot tests, it's easy as well. Two files. The
    comparison may be done automatically or manually.
    ... It's going to be part of the documentation we need to write for
    working groups.
    ... We also need to give guarantees that we're not going to break
    stuff like that.
    ... Back to your point, it means we need to make sure we have
    documentation for it. I'm at least working on that aspect for
    ... there will be different approaches to writing tests in different
    working groups.
    ... We're still learning, here.

Potential contributor

    Francois: Had a call with DKA about potential contributions for
    HTML5 tests
    ... they already open sourced their test suite

    l fd email on webtestsuite

      [10] http://lists.w3.org/Archives/Team/ercim-mobiweb/2011Mar/0054.html

    <plh-home> [11]http://webtestsuite.org/

      [11] http://webtestsuite.org/

    Francois: it's released until an Apache license
    ... their initial goal is to test WAC
    ... about 900 test cases. Some of them test performance imposed by
    WAC specs
    ... in the end, there are 200 tests for us
    ... HTML5, CSS, APIs, widgets, ...
    ... the tests are defined in JS
    ... they know that just open sourcing isn't enough

    plh: would be good to have a documentation already. I guess the best
    way to move forward is to tell them to join the task force, point
    them to the test harness.
    ... The bottom line is that we have this Mercurial server where
    people should drop their test cases that need to undergo a review
    ... It's not perfect.
    ... one requirement we have is to keep the tests separated.

    francois: ok, I'll get back to them.

    plh: one possibility is to have some code that generate test cases
    that we can integrate.

    francois: right, that's one possibility we mentioned

    plh: we don't want to have tests depend on the higher-level testing
    framework, because such test suites tend to die in the long run


      [12] http://w3c-test.org/webperf/tests/approved/test_navigation_attributes_exist.htm

    plh: here's an example of a JavaScript test.
    ... It contains several tests, and reports. It's a test. It's
    isolated, does not depend on a higher level framework.
    ... We're using the test harness, but that's, to some extent,
    orthogonal to the thing we're testing.
    ... The higher level test framework will collect all the results
    from all these tests and report them in some nice view or report.
    ... One possible problem with Webtestsuite tests is that they are
    dependent on the higher level harness, so cannot be taken away from


      [13] http://w3c-test.org/html/tests/approved/video/video_000.htm

    plh: another example. No JavaScript involved. You have to read the
    test to understand whether the test passed or failed. It's up to the
    framework to provide some means to the user to indicate whether the
    test passed or failed.
    ... It's not in the test itself.
    ... The test is independent on the framework.
    ... So if we decide to rewrite the framework in 10 years, it's fine.

    shadi: many accessibility tests will be more of the manual type.
    ... these examples are making it much more concrete, so thanks for
    ... I think Michael and I should be looking at accessibility tests
    and see if they fit under that categorization.
    ... I would also encourage your groups to write a few early tests
    that you think should be automatic and submit them to the HTML
    testing suite TF.
    ... Another example is a "ref test".


      [14] http://w3c-test.org/html/tests/submission/W3C/bidi-markup-export/html5-html/

    plh: The two files are supposed to render exactly the same way. If
    we cannot do that automatically, we can ask a human to report.
    Another way is to do screenshots.
    ... We're not entirely sure if it's doable, or not.

    <gsnedders> Each (major) browser has their own API for doing so,
    though I don't think IE has anything public, but otherwise do-able
    with per-browser harnesses.

    [small discussion on ref tests vs. DOM comparison tests]

Summary of Action Items

    [NEW] ACTION: plh to add examples to the Wiki in terms of testing
    methods [recorded in

    [End of minutes]

     Minutes formatted by David Booth's [16]scribe.perl version 1.135
     ([17]CVS log)
     $Date: 2011/03/22 13:09:41 $

      [16] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
      [17] http://dev.w3.org/cvsweb/2002/scribe/
Received on Tuesday, 22 March 2011 13:23:19 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:20:32 UTC