- From: Maciej Stachowiak <mjs@apple.com>
- Date: Tue, 24 Aug 2010 05:06:50 -0700
- To: James Graham <jgraham@opera.com>
- Cc: Kris Krueger <krisk@microsoft.com>, "'public-html-testsuite@w3.org' (public-html-testsuite@w3.org)" <public-html-testsuite@w3.org>
On Aug 24, 2010, at 3:48 AM, James Graham wrote: > On 06/24/2010 01:25 AM, Kris Krueger wrote: >> In an effort to improve browser interoperability, Microsoft has submitted more HTML5 tests for XHTML5. >> Microsoft welcomes feedback on the tests if they are incorrect per the HTML5 specification. >> >> http://test.w3.org/html/tests/submission/Microsoft/ > > (Apologies I missed this email earlier). > > These tests are all manual tests. However they are typically not testing things that *require* manual testing. Given the high costs of running manual tests and the relative difficulty of adding them to automated regression testing systems, I believe this should be sufficient grounds to reject the tests. Also, it is important for Process reasons that compiling implementation reports is simple. Having even a small percentage of manual tests out of the tens of thousands of tests that HTML5 will require will constitute a significant burden on the production and update of implementation reports. Therefore I propose that we have a policy that manual tests be accompanied by an explaination of why the test _must_ be manual and cannot be implemented as either a javascript test or a reftest. +1 We follow this policy for the WebKit project's regression tests - if you submit a manual test, you have to explain why a fully automated test is impossible. Regards, Maciej
Received on Tuesday, 24 August 2010 12:07:25 UTC