MSE test suite update (was: RE: EME test suite)

Hi all,
 
Philippe invited me to take a look at the Media Source Extensions spec, so I read the spec, evaluated existing tests, wrote a few additional test cases, and tried to list pending actions. The result is summarized in:
http://tidoust.github.io/media-source-testcoverage/
 
The "action plan" is my own understanding of what remains to be done, and of whom proposed or seems best positioned to do it:
http://tidoust.github.io/media-source-testcoverage/#action-plan
 
Do not read too much into the percentages that appear in the "covered" column. They are rough estimations. I did not evaluate SourceBuffer algorithms. They are implicitly checked whenever a test case issues an "appendBuffer" command but a few additional test cases are most likely required to check specific error conditions and more advanced scenarios.
 
I generated an updated test report with the latest versions of browsers that I have at hand. This report includes the test cases that I submitted so far, which have not yet been reviewed and integrated in the official web-platform-tests repo:
 
http://tidoust.github.io/media-source-testcoverage/results/all.html
http://tidoust.github.io/media-source-testcoverage/results/less-than-2.html
http://tidoust.github.io/media-source-testcoverage/results/complete-fails.html
 
In this report:
- CH53 stands for Chrome Canary 53.0.2773.0
- ED14 stands for Edge Insider Preview (Slow) build 14366
- FF49 stands for Firefox Developer Edition 49.0a2
 
I only have access to Safari for iOS here, and it does not seem to support MSE. Safari for Mac OS should support MSE though and should be added to the report. I'll try to find a Mac around. If anyone wants to give it a try in the meantime and send me the resulting JSON, the test suite I've been using is at:
https://github.com/tidoust/web-platform-tests/tree/media-source
(note it is in the "media-source" branch, not "master")
 
Note that a number of test files (but not necessarily the test cases within these test files) fail with a timeout in Edge. I think this is because Edge does not yet support resetting the delaying-the-load-event-flag, and thus does not fire the "load" event on window, which testharness uses to detect timeouts, unless "endOfStream()" is called.
 
I will join the call tomorrow.
 
Thanks,
Francois.
 
 
De : Paul Cotton [mailto:Paul.Cotton@microsoft.com] 
Envoyé : jeudi 16 juin 2016 19:16
À : Mark Watson <watsonm@netflix.com>; David Dorwin <ddorwin@google.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>; Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; fd@w3.org
Cc : public-hme-editors@w3.org
Objet : RE: EME test suite
 
I would like to suggest that we have an Editors’, Chair and W3C Team meeting on Tuesday in the 8am PDT meeting slot to discuss where we stand on MSE and EME testing.  Once the we get on the same page I would recommend that we open up the discussion to the entire HME WG.
 
I will send out an agenda either today or tomorrow.  Please provide any updates on this thread for now.
 
/paulc
 
From: Mark Watson [mailto:watsonm@netflix.com] 
Sent: Tuesday, June 14, 2016 3:39 AM
To: David Dorwin <ddorwin@google.com>
Cc: public-hme-editors@w3.org
Subject: Re: EME test suite
 
That's good.
 
Which DRM vendors will be a able to stand up test servers for the period whilst we prepare the implementation report ? (I don't think we can test the whole specification with ClearKey alone).
 
The tests I am working on will have a simple abstract interface to plug in different server protocols on the client side.
 
I also wonder whether it might be acceptable for the purpose of the implementation report to have some browser-specific polyfill code that patches up any small non-compliances ? We would still be testing the "true" EME API, just an implementation of it that consists of a browser + polyfill. This seems to me to meet the requirement for demonstrating specification correctness, at least. What do you think ?
 
...Mark
 
On Fri, Jun 10, 2016 at 1:41 PM, David Dorwin <ddorwin@google.com <mailto:ddorwin@google.com> > wrote:
Great! I'll create a PR for the Blink tests early next week.
 
On Fri, Jun 10, 2016 at 1:38 PM, Mark Watson <watsonm@netflix.com <mailto:watsonm@netflix.com> > wrote:
FYI: I started work on the IDL tests. I will put them here <https://github.com/mwatson2/web-platform-tests> , encrypted-media branch.
 
I am also going to look as porting our simple EME tests across to the web-platform-tests framework.
 
...Mark