Re: testing framework Re: [HTMLWG] CR Exit Criteria redux

On 09/27/2012 11:46 AM, Karl Dubost wrote:
>
> Le 27 sept. 2012 à 03:32, James Graham a écrit :
>> I am extremly skeptical about using that framework.
>
> What would be a practical and reasonable setup/framework for testing (for you)?

The ideal scenario for me is the following:

1) Browser vendors already run W3C tests as part of their day-to-day 
testing for regressions (this is increasingly already true, but we are 
not quite there yet).

2) For all automatable tests we simply ask browser vendors to provide 
the results they got when they regression tested the build that we are 
considering for conformance purposes. Exactly how they do this would 
vary from vendor to vendor. At a technical level we might consider 
having a common submission format that can be imported into some sort of UI.

3) Any further manual testing, including manual tests with e.g. 
screenreader + browser combination, are carried out using any suitable 
test management system, either internally by browser vendors, or 
externally, as the group decides is appropriate (this might vary from 
test to test). For externally administered tests of this type, the 
existing W3C framework might be adequate.

Received on Thursday, 27 September 2012 09:59:33 UTC