MSE test suite status update

Hi all,

Philippe invited me to contribute to the Media Source Extensions test suite effort. I evaluated existing tests, and tried to come up with an action plan, which we discussed yesterday with the MSE Editors and Chair. The evaluation and action plan are summarized in:

http://tidoust.github.io/media-source-testcoverage/

We expect to make progress on the action plan within the next couple of weeks.

Participants of the HME WG members are invited to take a look at this document and provide feedback as needed. Do not read too much into the percentages that appear in the "covered" column. They are rough estimations. The SourceBuffer algorithms still need to be properly evaluated. They are implicitly checked whenever a test case issues an "appendBuffer" command but additional test cases are most likely required to check specific error conditions and more advanced scenarios.

Philippe and I started to write additional test cases. I generated an updated implementation report with the latest versions of browsers that I have at hand that includes the test cases that Philippe and myself submitted so far, which still need to be reviewed and integrated in the official GitHub repository (the document mentioned above links to the related pull requests):

http://tidoust.github.io/media-source-testcoverage/results/all.html
http://tidoust.github.io/media-source-testcoverage/results/less-than-2.html
http://tidoust.github.io/media-source-testcoverage/results/complete-fails.html

In this report:
- CH53 stands for Chrome Canary 53.0.2774.0 launched with the "--enable-experimental-web-platform-features" flag
- ED14 stands for Edge Insider Preview (Slow) build 14366
- FF50 stands for Firefox Nightly 50.0a1 (2016-06-21)

Safari for Mac OS should be added to the report, which I will do that as soon as I get hold of a Mac.

As of today, main failures across browsers seem to be:
- TextTrack related features
- TrackDefault (feature marked at-risk)
- live seekable range
- on{event} attributes
- audioTracks/videoTracks attributes 
- A few steps that should now fire TypeError but still fire InvalidAccessError exceptions

That list may need to be completed as new tests are added to the test suite. For instance, appendStream() (also marked at-risk) is not yet tested and not supported.

Thanks,
Francois.

Received on Wednesday, 22 June 2016 09:17:49 UTC