- From: James Graham <jgraham@opera.com>
- Date: Tue, 20 Oct 2009 16:28:23 +0200
- To: Kris Krueger <krisk@microsoft.com>
- CC: "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>
Kris Krueger wrote: > 1.) Creation of an Informal Charter The deliverable will be the > creation of a document that is created and approved by the HTML WG > chairs that sets the goals and expectations for the testing task > force. I think it is important that we inherit the ability for asynchronous participation from the HTMLWG charter. That is any decisions must allow for asynchronous participation. In general I don't see a great need for synchronous communication in developing a testsuite so I think we should avoid putting any requirement for regular telecons or meetings or whatever in the charter. Other than that, I guess the goal is to create a testsuite for HTML5 sufficient to get us to REC. I don't know what else is needed. > 2.) Test Suite Organization The idea here is to enumerate and map out > a plan for how the test suite appears on the w3c site and how the > test suite supports the HTML5 specification. > > 3.) Test Case Methodology and Application With the wide range of > features in the HTML5 specification, having a consistent manner to > test 'similar' features of the specification should produce a higher > quality test suite. > > 4.) Test Case Submission, Review, Updates, Publishing, Versioning The > goal is to have a clear plan and process in place for helping get > tests created by a volunteer to the official test suite. For all of these points, I think we should be working closely with the existing w3c automated testing project [1] to decide how to structure our testcases. In particular we should look for testcases to be contributed in a format that is suitable for automation in the W3CBrowserRunner harness that they have developed. So, for example, tests for rendering should use RefTests where possible, javascript tests should use one of the supported javascript testing frameworks, and so on. In the exceptional case that it is not possible to use one of the existing frameworks for some reason we should ensure that support for whatever framework is provided is added to the test runner. Manual tests should be avoided. More specifically, I would expect us to break tests up by feature, possibly with several different testsuites for each feature reflecting different contributions from implementors. > 5.) Leveraging/Research of existing HTML5 Tests Undoubtedly various > browser vendors and independent individuals have potential test > collator that could be leveraged by the test suite. Yes, realistically I would expect most of our testcases to be donated by browser vendors when they implement a feature. So it is important to make this process as easy as possible subject to the constraints above (afaict this also dictates the licensing terms of contributed tests [2]) [1] http://omocha.w3.org/wiki/ [2] http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright
Received on Tuesday, 20 October 2009 14:28:18 UTC