RE: HTML Testing Task Force Conf Call Agenda 5/4/2010 - minutes

The minutes are available here: http://www.w3.org/2010/05/04-htmlt-minutes.html

- DRAFT -
HTML Weekly Teleconference
04 May 2010

Attendees
Present
  Plh, +04613479aaaa, gsnedders, adrianba, krisk
Chair
  krisk
Scribe
  adrianba

Contents

Topics 
1.Check for any bugs on approved tests (currently zero)
2.Review Current Tests Posted To List For Approval
3.Ask group for any upcoming tests to be approved/submitted
4.Any other business

Summary of Action Items

--------------------------------------------------------------------------------

Check for any bugs on approved tests (currently zero)
krisk: currently zero, but we can have some bugs once we make changes to the harness
... to support james graham's async feedback

gsnedders: don't want to start with one harness and to keep hacking another on top and one on top of that

krisk: do you have an idea what that should look like?

gsnedders: not really

<gsnedders> jgraham: ping

krisk: in the windows tests i think i did something along those lines

plh: i wonder how independent we can be from the harness
... so that the tests can be written in a way that makes it easy to switch to a new harness
... if we get another harness that is better
... we don't have to re-write everything

<jgraham> FWIW I think the async stuff is the stuff that makes it hard to switch harnesses too

krisk: is this about consistency so that we can depend upon certain things being in the test

gsnedders: the main thing is about having the assert functions that check for what the test is testing
... assertEquals, etc.

krisk: if you are too specific about what has to be written then you lose flexibility
... but it has to say pass/fail in a consistent manner

<jgraham> Given a set of assertFunctions and two harnesses that implement them, it should be quite easy to swicth synchronous tests between the two harnesses

krisk: there will be some tests where assertEquals isn't going to work for you

gsnedders: most harnesses have these kind of functions
... the main thing that varies is how you say what tests should be run

krisk: nothing would prevent you from changing later - for example i changed from jsunit by replacing a lot of the include files

<jgraham> but with an async test, you are typically deep into the harness-specific behaviour to determine what you need to write, when it gets called, how you express failure, and so on

krisk: in that regard, if a browser vendor didn't like how the tests ran they could make updates to run in their own way
... the basic contract between the test and the harness is all that has to be defined
... even if you had an async test you could poll and know the test was starting to run
... and through some setTimeouts continue to check if the work has been done
... and at some point the harness can say the work didn't complete and move on

<jgraham> That sounds like a harness-specific design

krisk: but in the end it needs to look for a pass or test completed result in a consistent way

<jgraham> You can't just replace a test designed to work in that way with one designed to work in a different way

plh: right now, kris you provided one?

krisk: i wouldn't say a harness, more like a loader
... could be extended to do more

plh: do we need to do more? i suggest we don't need to
... once we have plenty of tests we can focus on that if necessary

gsnedders: once we have plenty of tests it will be harder to change
... for example if someone gives lots of tests for a section of the spec

<jgraham> FWIW my point of view is that we should write a harness and provide hooks for people to get the results out for whatever purpose they have

<jgraham> e.g. customs regression-detection systems

<jgraham> *custom

plh: the reason i'm not in a hurry to develop another test harness is to avoid duplicating work
... would rather delay discussion on test harness until june
... see where we are then

krisk: regardless of the harness we should define the consisten way of seeing a pass

plh: yes, we should define assertEquals, etc

krisk: i mean even like tag id

gnsedders: why do we need to define that?

krisk: a number of tests have different ways of saying the pass

<jgraham> one assert != one passing test

krisk: at the moment mostly it is reading what is on the screen

gsnedders: i sent e-mail an hour ago about getting this programatically get this out

<krisk> this post ?

<krisk> http://lists.w3.org/Archives/Public/public-html-testsuite/2010May/0005.html

<gsnedders> yeah

krisk: when you call assertEquals, the implementation of that will go call something else
... so you take the logging out of the test
... then you can change assertEquals to do whatever you want

<jgraham> So one approach is that you have a callback funtion that you call when the test has a result

gsnedders: this is what most of the JS libraries do

krisk: otherwise you have a consistent place - the harness knows to go look there

gsnedders: the problem is that you have to keep polling the DOM
... but a function means you can send it back as soon as you get the call
... makes it easy to send back to regression tracking systems

<jgraham> To be clear, a single assertEquals should not be considered a single result. One result may be the combination of several assertions

krisk: should the harness be in control? what happens if the test blows up?

gsnedders: the harness should catch all exceptions
... for critical issues in a browser you won't be able to access the DOM either
... i don't think there's a difference

krisk: it sounds like we would want to change the DOM tests JS implementation of assertEquals to fit into a harness at some point

gsnedders: yeah

krisk: sounds like we have agreement

<krisk> wiki -> http://www.w3.org/html/wg/wiki/Testing

<plh> http://www.w3.org/html/wg/wiki/Main_Page

Review Current Tests Posted To List For Approval
krisk: we've covered this too

Ask group for any upcoming tests to be approved/submitted
krisk: is anyone thinking about submitting more tests?

plh: one of the reasons I was asking this - do we have anything that says which areas we are looking for tests for
... for example, progress events for video aren't implemented in some browsers - if we ask for video tests we might get tests for things that aren't stable
... i remember we said that we wanted tests for window because that was more stable

krisk: video would be one

plh: yes, as long as you don't go too far
... for example, testing the js play function is fine
... i might start working on video tests, probably not in may

<jgraham> Writing video tests is one way to ensure that we support async tets well :)

<jgraham> *tests

krisk: some people at microsoft have some tests that we'd like to submit soon too

plh: i don't want to duplicate effort
... if there are people who will submit video tests soonish

gsnedders: i will when i have time
... hopefully this month

plh: video will be important - <video> gets the most buzz

krisk: do we want to use <video> to test the async part of our requirements?

plh: we could

krisk: we should

gsnedders: okay

Any other business
<plh> http://lists.w3.org/Archives/Public/public-html-testsuite/2010Apr/0017.html

plh: sent a report last week about getting things moved to the W3C site
... after discussing with system folks they would like to avoid putting thousands of tests into main site
... running into issues with css2.1
... will provide access to a web site by end of may
... will have to different hosts, two different domains in fact
... so that we can do cross-origin tests if needed
... also looking at whether test harness runs on this server or if we need it on a third server

<plh> tests.w3.org and tests.www.org

<krisk> for example....

<krisk> tests.w3.org/html5/

krisk: i'd make one primary - i think one ending in w3.org is important

plh: these sites will also host the css2.1
... would prefer not to run php or whatever on the same site
... prefer to run harness somewhere else
... but it's not possible to differentiate which is the harness at the moment
... still considering how to approach the problem
... but if harness only uses js then not an issue
... only an issue if we want to run server-side php or python

gsnedders: we will want some tests eventually for, e.g. slow loading images

plh: at some point we will need something on the server-side but not in the immediate future
... so we should run with this idea by end of may and this is the main priority right now
... expect more information next week
... not going to get deployment right away because of that but will have a better solution by the end of the month

krisk

<plh> htmt5tests.w3.org

krisk: i think this is a good starting point
... keep it simple in the short term seems good to me
... anything else?

<krisk> When I push to HG now i get a permission denied error

<krisk> I hit the error on 2 differnet systems

<krisk> specific error

<krisk> permission denied: .hg/store/lock

gsnedders: for the video tests, we're going to have to use some video format

plh: we can use two

gsnedders: yes, we can use two

<plh> http://www.w3.org/2008/12/dfxp-testsuite/web-framework/START.html

plh: this is similar to what happened in the timed text working group
... there is a video and it is available in multiple formats
... it's not a big deal
... we could use the same videos - they were done for the purpose of testing the timing
... but it doesn't contain sound

gsnedders: we have test videos too

plh: yes, if people want to submit videos as long as they have the rights we can use them
... and we can transform into whatever format we need to

krisk: sounds like we have agreement that we'll have to support different formats
... and plh will help with converting

plh: we will run into some problems for example testing the src attribute
... but we could have some JS hacks testing which the browser supports

krisk: any other business?
... okay, meeting adjourned

Summary of Action Items
[End of minutes]

Received on Tuesday, 4 May 2010 15:58:09 UTC