W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2011

Notes from 2 November 2011 #testing breakout

From: Arthur Barstow <art.barstow@nokia.com>
Date: Wed, 02 Nov 2011 13:41:29 -0700
Message-ID: <4EB1AAF9.8010707@nokia.com>
To: public-test-infra <public-test-infra@w3.org>
Here are the IRC notes from the November 2 #testing breakout (copied below):

http://www.w3.org/2011/11/02-testing-minutes.html

(I've made a couple of slight editorial changes in the text copy below.)

-ArtB

    [1]W3C

       [1] http://www.w3.org/

                                - DRAFT -

                      #testing breakout @ TPAC 2011

02 Nov 2011

    See also: [2]IRC log

       [2] http://www.w3.org/2011/11/02-testing-irc

Attendees

    Present
           Wilhelm, PeterL, DavidB, JamesG, PLH, KrisK, ArtB, Laszlo,
           EdO, MarkV, MikeSmith, JohnJ, JacqueD, JanL, MattW, CyrilC,
           Russell

    Regrets
    Chair
           James

    Scribe
           ArtB

Contents

      * [3]Topics
      * [4]Summary of Action Items
      _________________________________________________________

    <scribe>  Chair: James

    Date: 2 November 2011

    <scribe>  Scribe: ArtB

    <scribe>  Agenda: TPAC Testing - a Practical Session

    JG: welcome everyone
    ... start with current state

     in the various WGs

     I know about WebApps and HTML WGs

     want to look at testing formats

     and the procs for gathering tests

     Would like to know about areas for improvement

    KK: how many are familiar with Hg/Mercurial?

     and submitting tests?

     And how to create W3C tests?

     [ Not very many ]

     Can talk about how to push test cases to W3C

     We don't @ Msft use Hg

     but it's pretty easy

     Can search for Mercurial

    <JohnJansen>  [5]http://mercurial.selenic.com/

       [5] http://mercurial.selenic.com/

    AB:
    [6]http://www.w3.org/2008/webapps/wiki/WorkMode#CVS_and_Mercurial

       [6] http://www.w3.org/2008/webapps/wiki/WorkMode#CVS_and_Mercurial

    KK: the HTML WG has a wiki of resources

     we use public-html-testsuite

    <stearns>  [7]http://wiki.csswg.org/test/mercurial

       [7] http://wiki.csswg.org/test/mercurial

     HTML wiki: [8]http://www.w3.org/html/wg/wiki/Testing

       [8] http://www.w3.org/html/wg/wiki/Testing

     after getting Mercurial

     must create a ~/.hgrc file

    <jgraham>  [9]http://www.w3.org/html/wg/wiki/Testing

       [9] http://www.w3.org/html/wg/wiki/Testing

     or ini file for other OS's

     May need to mess with proxy settings if behind a f/w

     after install, Hg should be in $PATH

     the verbs are pull and push

     to get a local copy, use pull

    <krisk>  [10]http://www.w3.org/html/wg/wiki/Testing/Submission/

      [10] http://www.w3.org/html/wg/wiki/Testing/Submission/

    <jgraham>  [11]http://dvcs.w3.org/hg/

      [11] http://dvcs.w3.org/hg/

    AB: mirror is: [12]http://w3c-test.org//

      [12] http://w3c-test.org//

    KK: the Hg root has tests and specs

     so not just about tests

     after a test is `push`ed to server, it is stored in a src control
    system

     can do some complex stuff on backend e.g. PHP

    <plh_>  -->  [13]http://www.w3.org/wiki/TestInfra/goals Test
    infrastructure goals

      [13] http://www.w3.org/wiki/TestInfra/goals

     Several WGs are using it

     there is a `resources` folder

     it includes some sample tests

    <jgraham>  [14]http://w3c-test.org/resources/

      [14] http://w3c-test.org/resources/

    JG: there are some sample HTML5 API tests

     e.g. [15]http://w3c-test.org//resources/apisample.htm

      [15] http://w3c-test.org//resources/apisample.htm

     designed to integrate with internal testing
    resources/frameworks/tools

     can make them talk to each other a bit

    KK: expect people to pull tests and run them internally

     W3C server can't support lots of browser vendors running their
    tests on w3.org resources

    Q: is it possible to select specific test cases for a spec?

     so not clear if I can find specific test cases

    JG: for HTML test suite, the directory names give a hint

     the CSS WG is working on something more sophisticated

    PL: we are working on a bug tracker for the tests

     want to support test case management

     including metadata

     want to link test cases to specific parts of the spec

     it is up and running now

     but still needs some work

    Q: how about optional vs. mandatory parts of the spec?

    PL: we want to support that

     we need to markup the spec to facilitate test cases pointing to
    specific parts of the spec

     If want to browse the spec, want to be able to see what tests are
    available

     We also use an annotation system

     so that results from tests runs are available to someone browsing
    the spec

    Mark: is there any type of overall plan?

     and some data about coverage?

     Also, is there some convergence for tools?

    JG: yes, there is work toward tool convergence

     at least at the harness level

    JG: re the 1st question, that's pretty complicated

    Mark: want to understand the plan for ex, HTML5

    KK: we try to facilitate, but no hard rules

     it's up to the WG participants re what will actually get done

     We have a structure to enable lots and lots of tests

     We have some features that have lots (1,000s) of tests and other
    features with more like 10's of tests

    Mark: for CSS 2.1, is test suite done?

    PL: one reason for the tests is to get to REC

     the other reason is to determine if everyone implements the spec
    correctly

     we try to use a s/w development process

    Mark: are there specific milestones?

    PL: not really, we continue to improve

    DB: as bugs are found, new tests are added

    PL: as we hit milestones, we snapshot the test suite

     and add new tests if we need to

    KK: we need to get a handle on how close the test suite is to the
    impls

     and that can be hard to determine

     F.ex, web workers is already supported in several browsers

     and mostly interoperable so there is a question about how much
    testing effort should be done

    <plh_>  -->  q?

     The work done depends on the WG participants

    PL: when a spec is early, it's a bit iffy to write tests

     but it would be useful too

    JG: eventually must write tests so it makes sense to start early

    LG: agree converging test frameworks is important

     need to eliminate outside dependencies

    KK: yes, we ran into that with some old DOM tests (from NIST)

    JG: we have a framework now

     that is getting convergence

     it is platform independent

     it does depend on JavaScript

     but no DOM dependencies

    KK: we need to be careful about adding features to the framework

    AB: how many WGs are using testharness?

    PLH: HTML, WebApps, Web Performance, Web Events

    John: it was relatively easy for some tests

    LG: I think DAPI WG agreed to use testharness

     but they don't have many tests yet

    KK: I think we need to make sure WGs use testharness

    JD: what about rendering tools?

    JG: there are RefTests and manual tests

     testharness is a JS api

     reports results via a callback

    PLH: we have a framework on the test server to collect test results
    (HTML WG)

    PL: we can import results from other formats

     the format can be XML or whatever

     we also have an XHR API

    JD: need a good format

    John: testharness is really for JS APIs

     there are challenges for visual matches

    KK: yes, e.g. font variability

    s/Q?/Jan/g

    KK: if there are any questions, please send them to
    public-test-infra

     that's the list for the Testing IG

    [ James gives a quick tutorial on how to create a test with
    testharness.js ]

    Russ: what about images?

    JG: most use RefTests

     and then can compare rendering

    KK: the biggest problem with rendering is fonts

     RefTests give expected output

     tests with "ref" in the test file name

     at runtime, if they render the same, the test passes

    DB: a test in RefTest is an assertion of pixel equality

     want to exclude external factors like fonts, margins, etc.

    <scribe>  Chair: James

Summary of Action Items

    [End of minutes]
Received on Wednesday, 2 November 2011 20:41:55 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 2 November 2011 20:41:55 GMT