Re: XHTML5 Test Submission

On 08/24/2010 05:12 PM, Philippe Le Hegaret wrote:
> On Tue, 2010-08-24 at 12:48 +0200, James Graham wrote:
>> On 06/24/2010 01:25 AM, Kris Krueger wrote:
>>> In an effort to improve browser interoperability, Microsoft has submitted more HTML5 tests for XHTML5.
>>> Microsoft welcomes feedback on the tests if they are incorrect per the HTML5 specification.
>>>
>>> http://test.w3.org/html/tests/submission/Microsoft/
>>
>> (Apologies I missed this email earlier).
>>
>> These tests are all manual tests. However they are typically not testing
>> things that *require* manual testing. Given the high costs of running
>> manual tests and the relative difficulty of adding them to automated
>> regression testing systems, I believe this should be sufficient grounds
>> to reject the tests. Also, it is important for Process reasons that
>> compiling implementation reports is simple. Having even a small
>> percentage of manual tests out of the tens of thousands of tests that
>> HTML5 will require will constitute a significant burden on the
>> production and update of implementation reports. Therefore I propose
>> that we have a policy that manual tests be accompanied by an
>> explaination of why the test _must_ be manual and cannot be implemented
>> as either a javascript test or a reftest.
>
> I would agree with that but we have no way to run reftests automatically
> at the moment. Our test harness doesn't support it.
>
> So, how would we use those reftests?

So the best approach is to use self-describing reftests that work either 
as manual tests or as reftests. Then if the harness doesn't support 
reftests they can be run as manual tests. In the short term we can rely 
on browser vendors to use their own reftest frameworks. In the longer 
term we can consider standardising some screenshot API that we can use 
to implement reftests in a standard way (although I guess there are 
security concerns there). Alternatively we might be able to add support 
for the various APIs that different browsers support to some test runner 
and select the right implementation depending on the browser being 
tested. In fact I think the W3CTestRunner project already has some 
featuress in this direction?

Received on Wednesday, 25 August 2010 09:32:16 UTC