Re: HTML WG Testing Task Force

Kris Krueger wrote:
> 1.) Creation of an Informal Charter The deliverable will be the
> creation of a document that is created and approved by the HTML WG
> chairs that sets the goals and expectations for the testing task
> force.

I think it is important that we inherit the ability for asynchronous 
participation from the HTMLWG charter. That is any decisions must allow 
for asynchronous participation. In general I don't see a great need for 
synchronous communication in developing a testsuite so I think we should 
avoid putting any requirement for regular telecons or meetings or 
whatever in the charter.

Other than that, I guess the goal is to create a testsuite for HTML5 
sufficient to get us to REC. I don't know what else is needed.

> 2.) Test Suite Organization The idea here is to enumerate and map out
> a plan for how the test suite appears on the w3c site and how the
> test suite supports the HTML5 specification.
> 
> 3.) Test Case Methodology and Application With the wide range of
> features in the HTML5 specification, having a consistent manner to
> test 'similar' features of the specification should produce a higher
> quality test suite.
> 
> 4.) Test Case Submission, Review, Updates, Publishing, Versioning The
> goal is to have a clear plan and process in place for helping get
> tests created by a volunteer to the official test suite.

For all of these points, I think we should be working closely with the 
existing w3c automated testing project [1] to decide how to structure 
our testcases. In particular we should look for testcases to be 
contributed in a format that is suitable for automation in the 
W3CBrowserRunner harness that they have developed. So, for example, 
tests for rendering should use RefTests where possible, javascript tests 
should use one of the supported javascript testing frameworks, and so 
on. In the exceptional case that it is not possible to use one of the 
existing frameworks for some reason we should ensure that support for 
whatever framework is provided is added to the test runner.

Manual tests should be avoided.

More specifically, I would expect us to break tests up by feature, 
possibly with several different testsuites for each feature reflecting 
different contributions from implementors.

> 5.) Leveraging/Research of existing HTML5 Tests Undoubtedly various
> browser vendors and independent individuals have potential test
> collator that could be leveraged by the test suite.

Yes, realistically I would expect most of our testcases to be donated by 
browser vendors when they implement a feature. So it is important to 
make this process as easy as possible subject to the constraints above 
(afaict this also dictates the licensing terms of contributed tests [2])


[1] http://omocha.w3.org/wiki/
[2] http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright

Received on Tuesday, 20 October 2009 14:28:18 UTC