Re: ACTION-71: Investigate testing status to move things forward

Hi Paul,

Here is what I did so far:
1) I took the current tests from the web-platform-tests repo [1] not 
including any of the pull requests. I ran those tests through the test 
runner (locally after setting it up on my machine with the help of Robin 
because the online runner [2] was not working), it produced the JSON 
results (for the browsers I had on my machine: Chrome, IE, Firefox) that 
I pushed to the test-results repo [3]. Now that the online test runner 
seems to be working, anyone should be able to update that status: just 
go to [2] enter "/media-source" in "Run tests under path" and hit 
"Start". Once the tests are done "Download JSON results" save it under a 
2-letter-2-digits name (such as Ch41.json, or Fx36.json, IE11.json ...) 
and push that to [3].

2) Then I started analyzing the existing tests and identified some 
problems with those related to isTypeSupported. I proposed some changes 
and made the associated pull request [4].

3) Finally, I started reading the spec (again) to identify which tests 
were needed and as you suggested at TPAC to get a sense of coverage of 
the spec by the tests. I came up with a tentative list of 378 tests. 
Since there are only 228 existing tests, I started looking at which ones 
were missing. I have not finished that work yet. The current status is 
here [5]. It's a spreadsheet with 2 parts: one is a list of tests 
organized according the sections of the spec, and the other one is a 
list of tests organized by the names of the files in the test repo. I 
started cross-referencing them but this takes quite a lot of time...

That's it for now. I hope that helps. I might be able to work again on 
the tests in January if my students and other projects don't keep me too 
busy.

Cyril

[1] https://github.com/w3c/web-platform-tests/tree/master/media-source
[2] http://w3c-test.org/tools/runner/index.html
[3] 
https://github.com/w3c/test-results/commit/265eaeb832a4deb19412dfa863fd422889263ffc
[4] https://github.com/w3c/web-platform-tests/pull/1405
[5] 
https://docs.google.com/spreadsheets/d/1XKjIuGWjEaSvMkf31HiTaYYkLbyuNg4saP2Sf-iW-cs/edit?usp=sharing

Le 15/12/2014 21:15, Paul Cotton a écrit :
>
> Can you provide an update on the status of your MSE testing efforts?
>
> Paul Cotton, Microsoft Canada
>
> 17 Eleanor Drive, Ottawa, Ontario K2E 6A3
>
> Tel: (425) 705-9596 Fax: (425) 936-7329
>
> *From:*Paul Cotton [mailto:Paul.Cotton@microsoft.com]
> *Sent:* Monday, November 17, 2014 7:47 PM
> *To:* cyril.concolato@telecom-paristech.fr
> *Cc:* public-html-media@w3.org; Robin Berjon (robin@w3.org)
> *Subject:* RE: ACTION-71: Investigate testing status to move things 
> forward
>
> > ACTION-71: Investigate testing status to move things forward
>
> https://www.w3.org/html/wg/media/track/actions/71
>
> I understand from Robin that he has spent some time helping you come 
> up to speed on the existing MSE tests and that you may have some 
> initial test results.
>
> Could you reply to this message with a summary of your status and 
> plans for MSE testing?
>
> I would also like to know if you have enough information that you 
> would agree that ACTION-71 can be closed?
>
> /paulc
>
> Paul Cotton, Microsoft Canada
>
> 17 Eleanor Drive, Ottawa, Ontario K2E 6A3
>
> Tel: (425) 705-9596 Fax: (425) 936-7329
>


-- 
Cyril Concolato
Multimedia Group / Telecom ParisTech
http://concolato.wp.mines-telecom.fr/
@cconcolato

Received on Tuesday, 16 December 2014 12:31:53 UTC