- From: Arthur Barstow <art.barstow@nokia.com>
- Date: Wed, 02 Nov 2011 13:41:29 -0700
- To: public-test-infra <public-test-infra@w3.org>
Here are the IRC notes from the November 2 #testing breakout (copied below):
http://www.w3.org/2011/11/02-testing-minutes.html
(I've made a couple of slight editorial changes in the text copy below.)
-ArtB
[1]W3C
[1] http://www.w3.org/
- DRAFT -
#testing breakout @ TPAC 2011
02 Nov 2011
See also: [2]IRC log
[2] http://www.w3.org/2011/11/02-testing-irc
Attendees
Present
Wilhelm, PeterL, DavidB, JamesG, PLH, KrisK, ArtB, Laszlo,
EdO, MarkV, MikeSmith, JohnJ, JacqueD, JanL, MattW, CyrilC,
Russell
Regrets
Chair
James
Scribe
ArtB
Contents
* [3]Topics
* [4]Summary of Action Items
_________________________________________________________
<scribe> Chair: James
Date: 2 November 2011
<scribe> Scribe: ArtB
<scribe> Agenda: TPAC Testing - a Practical Session
JG: welcome everyone
... start with current state
… in the various WGs
… I know about WebApps and HTML WGs
… want to look at testing formats
… and the procs for gathering tests
… Would like to know about areas for improvement
KK: how many are familiar with Hg/Mercurial?
… and submitting tests?
… And how to create W3C tests?
… [ Not very many ]
… Can talk about how to push test cases to W3C
… We don't @ Msft use Hg
… but it's pretty easy
… Can search for Mercurial
<JohnJansen> [5]http://mercurial.selenic.com/
[5] http://mercurial.selenic.com/
AB:
[6]http://www.w3.org/2008/webapps/wiki/WorkMode#CVS_and_Mercurial
[6] http://www.w3.org/2008/webapps/wiki/WorkMode#CVS_and_Mercurial
KK: the HTML WG has a wiki of resources
… we use public-html-testsuite
<stearns> [7]http://wiki.csswg.org/test/mercurial
[7] http://wiki.csswg.org/test/mercurial
… HTML wiki: [8]http://www.w3.org/html/wg/wiki/Testing
[8] http://www.w3.org/html/wg/wiki/Testing
… after getting Mercurial
… must create a ~/.hgrc file
<jgraham> [9]http://www.w3.org/html/wg/wiki/Testing
[9] http://www.w3.org/html/wg/wiki/Testing
… or ini file for other OS's
… May need to mess with proxy settings if behind a f/w
… after install, Hg should be in $PATH
… the verbs are pull and push
… to get a local copy, use pull
<krisk> [10]http://www.w3.org/html/wg/wiki/Testing/Submission/
[10] http://www.w3.org/html/wg/wiki/Testing/Submission/
<jgraham> [11]http://dvcs.w3.org/hg/
[11] http://dvcs.w3.org/hg/
AB: mirror is: [12]http://w3c-test.org//
[12] http://w3c-test.org//
KK: the Hg root has tests and specs
… so not just about tests
… after a test is `push`ed to server, it is stored in a src control
system
… can do some complex stuff on backend e.g. PHP
<plh_> --> [13]http://www.w3.org/wiki/TestInfra/goals Test
infrastructure goals
[13] http://www.w3.org/wiki/TestInfra/goals
… Several WGs are using it
… there is a `resources` folder
… it includes some sample tests
<jgraham> [14]http://w3c-test.org/resources/
[14] http://w3c-test.org/resources/
JG: there are some sample HTML5 API tests
… e.g. [15]http://w3c-test.org//resources/apisample.htm
[15] http://w3c-test.org//resources/apisample.htm
… designed to integrate with internal testing
resources/frameworks/tools
… can make them talk to each other a bit
KK: expect people to pull tests and run them internally
… W3C server can't support lots of browser vendors running their
tests on w3.org resources
Q: is it possible to select specific test cases for a spec?
… so not clear if I can find specific test cases
JG: for HTML test suite, the directory names give a hint
… the CSS WG is working on something more sophisticated
PL: we are working on a bug tracker for the tests
… want to support test case management
… including metadata
… want to link test cases to specific parts of the spec
… it is up and running now
… but still needs some work
Q: how about optional vs. mandatory parts of the spec?
PL: we want to support that
… we need to markup the spec to facilitate test cases pointing to
specific parts of the spec
… If want to browse the spec, want to be able to see what tests are
available
… We also use an annotation system
… so that results from tests runs are available to someone browsing
the spec
Mark: is there any type of overall plan?
… and some data about coverage?
… Also, is there some convergence for tools?
JG: yes, there is work toward tool convergence
… at least at the harness level
JG: re the 1st question, that's pretty complicated
Mark: want to understand the plan for ex, HTML5
KK: we try to facilitate, but no hard rules
… it's up to the WG participants re what will actually get done
… We have a structure to enable lots and lots of tests
… We have some features that have lots (1,000s) of tests and other
features with more like 10's of tests
Mark: for CSS 2.1, is test suite done?
PL: one reason for the tests is to get to REC
… the other reason is to determine if everyone implements the spec
correctly
… we try to use a s/w development process
Mark: are there specific milestones?
PL: not really, we continue to improve
DB: as bugs are found, new tests are added
PL: as we hit milestones, we snapshot the test suite
… and add new tests if we need to
KK: we need to get a handle on how close the test suite is to the
impls
… and that can be hard to determine
… F.ex, web workers is already supported in several browsers
… and mostly interoperable so there is a question about how much
testing effort should be done
<plh_> --> q?
… The work done depends on the WG participants
PL: when a spec is early, it's a bit iffy to write tests
… but it would be useful too
JG: eventually must write tests so it makes sense to start early
LG: agree converging test frameworks is important
… need to eliminate outside dependencies
KK: yes, we ran into that with some old DOM tests (from NIST)
JG: we have a framework now
… that is getting convergence
… it is platform independent
… it does depend on JavaScript
… but no DOM dependencies
KK: we need to be careful about adding features to the framework
AB: how many WGs are using testharness?
PLH: HTML, WebApps, Web Performance, Web Events
John: it was relatively easy for some tests
LG: I think DAPI WG agreed to use testharness
… but they don't have many tests yet
KK: I think we need to make sure WGs use testharness
JD: what about rendering tools?
JG: there are RefTests and manual tests
… testharness is a JS api
… reports results via a callback
PLH: we have a framework on the test server to collect test results
(HTML WG)
PL: we can import results from other formats
… the format can be XML or whatever
… we also have an XHR API
JD: need a good format
John: testharness is really for JS APIs
… there are challenges for visual matches
KK: yes, e.g. font variability
s/Q?/Jan/g
KK: if there are any questions, please send them to
public-test-infra
… that's the list for the Testing IG
[ James gives a quick tutorial on how to create a test with
testharness.js ]
Russ: what about images?
JG: most use RefTests
… and then can compare rendering
KK: the biggest problem with rendering is fonts
… RefTests give expected output
… tests with "ref" in the test file name
… at runtime, if they render the same, the test passes
DB: a test in RefTest is an assertion of pixel equality
… want to exclude external factors like fonts, margins, etc.
<scribe> Chair: James
Summary of Action Items
[End of minutes]
Received on Wednesday, 2 November 2011 20:41:55 UTC