W3C home > Mailing lists > Public > public-html-testsuite@w3.org > August 2010

Re: XHTML5 Test Submission

From: Maciej Stachowiak <mjs@apple.com>
Date: Tue, 24 Aug 2010 05:06:50 -0700
Cc: Kris Krueger <krisk@microsoft.com>, "'public-html-testsuite@w3.org' (public-html-testsuite@w3.org)" <public-html-testsuite@w3.org>
Message-id: <17157C7E-B8DB-4D26-BD6D-5BB8301272AD@apple.com>
To: James Graham <jgraham@opera.com>

On Aug 24, 2010, at 3:48 AM, James Graham wrote:

> On 06/24/2010 01:25 AM, Kris Krueger wrote:
>> In an effort to improve browser interoperability, Microsoft has submitted more HTML5 tests for XHTML5.
>> Microsoft welcomes feedback on the tests if they are incorrect per the HTML5 specification.
>> 
>> http://test.w3.org/html/tests/submission/Microsoft/
> 
> (Apologies I missed this email earlier).
> 
> These tests are all manual tests. However they are typically not testing things that *require* manual testing. Given the high costs of running manual tests and the relative difficulty of adding them to automated regression testing systems, I believe this should be sufficient grounds to reject the tests. Also, it is important for Process reasons that compiling implementation reports is simple. Having even a small percentage of manual tests out of the tens of thousands of tests that HTML5 will require will constitute a significant burden on the production and update of implementation reports. Therefore I propose that we have a policy that manual tests be accompanied by an explaination of why the test _must_ be manual and cannot be implemented as either a javascript test or a reftest.

+1

We follow this policy for the WebKit project's regression tests - if you submit a manual test, you have to explain why a fully automated test is impossible.

Regards,
Maciej
Received on Tuesday, 24 August 2010 12:07:25 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 24 August 2010 12:07:25 GMT