Re: Testing getUserMedia

On Jun 1, 2012, at 3:35 AM, Dominique Hazael-Massieux wrote:

> Le jeudi 31 mai 2012 à 08:20 -0600, Cullen Jennings a écrit :
>> Two side questions on testing ...
>> 
>> Could we use a repo in the W3C area on github (https://github.com/w3c) instead of mercurial ?
> 
> There is actually integration between the mercurial repository and our
> test framework http://w3c-test.org/framework/ so that wouldn't be
> entirely trivial.
> 
> That being said, I know that other groups have a dual presence on github
> and dvcs.w3.org, and it might be that in general we're better served by
> having this situation as well.
> 
> I assume you're asking for github to make it easier to contribute to the
> test suite?

that's sort of what I was wondering about but it sounds like the integration with hg makes that less desirable. I'm happy to go with what you know works. 

> 
>> Could you include in the test some sort of automation tool so that the text can be run in an automated form ?
> 
> That's already the case; the reason for including testharness.js and
> testharnessreport.js in each test case is precisely to make automation
> possible; see for instance how Mozilla are using it:
> https://bugzilla.mozilla.org/show_bug.cgi?id=647323

thanks

> 
> Dom
> 
>> 
>> On May 31, 2012, at 4:35 AM, Dominique Hazael-Massieux wrote:
>> 
>>> Hi,
>>> 
>>> As getUserMedia matures and is getting deployed, the need to ensure
>>> interoperability across implementation increases; the only right way to
>>> ensure that interoperability is to have tests.
>>> 
>>> While formally we only need to show interoperability during Candidate
>>> Recommendation, I think it's worthwhile starting to create tests now,
>>> even if that means that some of these tests will have to modified to
>>> keep up with changes in the spec.
>>> 
>>> This message tries to serve as a general intro to how we do testing at
>>> W3C.
>>> 
>>> At a high level, a test case for a JavaScript API is an HTML file that
>>> exercises a specific aspect of the API and tries to determine if the API
>>> behaves as specified or not when run in the browser under test.
>>> 
>>> W3C groups working in this space use a common framework to develop test
>>> cases that facilitate automating the run of these test cases, as well as
>>> the collection of results from browsers running them. That test harness
>>> is described at:
>>> http://www.w3.org/2008/webapps/wiki/Harness
>>> 
>>> Unless there is a strong reason not to, I think we too should adopt that
>>> harness for the development of our test cases.
>>> 
>>> Process-wise, I think we should also follow the way of other groups:
>>> * have someone in the group designed as the test facilitator, that
>>> ensures that test cases get submitted, reviewed, approved
>>> 
>>> * test cases should be submitted either by email or better by uploading
>>> them to a dedicated mercurial repository; I've created
>>> https://dvcs.w3.org/hg/media-capture/file/tip to that end, to which
>>> anyone in DAP and WebRTC should have read-write access
>>> 
>>> * test cases are first put into the "submitted" directory; they'll get
>>> moved to "approved" once the group gets a chance to review and approve
>>> them
>>> 
>>> * we can also accept contribution of test cases from non group
>>> participants; I can explain more about the logistics of this when needed
>>> 
>>> We probably need to define how we want to review and approve test cases;
>>> different groups have had different approaches. But that's probably
>>> easier done once we have found a test facilitator for the spec :)
>>> 
>>> I've started creating test cases which I hope can also serve as useful
>>> starting points for other contributors; I'll give more details about
>>> this in a separate mail.
>>> 
>>> Dom
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 
>

Received on Friday, 1 June 2012 14:52:47 UTC