RE: [Agenda] HTML Media Extensions WG Editors meeting, Tue Jul 19

I’m not completely opposed to doing polyfills, if Paul and Phillippe think it’s useful; but don’t much like the idea of patching implementations in JS.  Waiting in CR and getting real implementation results seems much preferred.  If that is not an option, then I’d much prefer reviewing the gaps and seeking assurances that browsers intend to implement.  I think we said that was a last resort option though.

Jerry

From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Wednesday, July 20, 2016 11:04 AM
To: David Dorwin <ddorwin@google.com>
Cc: Matt Wolenetz <wolenetz@google.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>; Francois Daoust <fd@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>
Subject: Re: [Agenda] HTML Media Extensions WG Editors meeting, Tue Jul 19



On Wed, Jul 20, 2016 at 10:48 AM, David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> wrote:
For the purposes of the implementation report, I think we should just clearly indicate which features/steps are not passing and reference a bug report acknowledged/confirmed by the UA vendor.

​I think in that case we would find a large number of features that are not supported by two implementations. Usually one would respond to this situation by waiting a while in CR until implementations catch up, but we do not have that option here.

On the call yesterday, we agreed that it would be useful to also provide the Director with information about those features which can easily be patched up with JS to pass the tests. For the purpose of indicating the maturity / interoperability​ / design of the API I don't see that it makes much difference whether you are testing a C++ implementation or a (C++/JS) implementation.


In general, I think smaller targeted tests will help - allowing most tests to pass while clearly identifying specific issues. For larger tests, one option is to break them up. Alternatively, if there is a single part of a test that is failing and the UA can't be fixed in time, I suggest temporarily a) commenting that part of the test out if possible and b) adding an explicit test for that functionality.

The specific bug you cite has been fixed in ToT for a couple weeks (the status was stale), so Chrome Canary (54) should pass the test.

On Tue, Jul 19, 2016 at 7:21 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
All,

Regarding also presenting test results using polyfills, how could we proceed with this logistically ?

For example, the drmtoday test case I just submitted fails on Chrome, but only because of this bug<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> regarding a missing keystatuseschange message on close. I have a small polyfill file which patches this up and then the test passes.

Should we have a way to run the tests with / without polyfills like this ? Anyone have an idea what that mechanism should be ?

...Mark

On Tue, Jul 19, 2016 at 3:56 PM, Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> wrote:
Jerry, et al, I have 11 pending media-source v1Editorial pull requests pending review currently: https://github.com/w3c/media-source/pulls?q=is%3Aopen+is%3Apr+author%3Awolenetz+sort%3Acreated-asc


As these build on each other, and also as some of them update the byte stream format registry (and bytestream format specs), please review these ASAP so they can land and give plh@ time to subsequently
fix (while he's available this week) https://github.com/w3c/media-source/issues/74 without causing major merge conflicts/etc.

Thanks,
Matt

On Mon, Jul 18, 2016 at 3:21 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:

W3C EME Test Status



Overview:

Many thanks to the contributions of Greg Rutz (CableLabs) and Michael Stattmann (CastLabs) for their assistance in developing a test framework for browser and CDM multi-DRM interoperability.



The planned approach is:

1.           Evaluate current test coverage: Determine the coverage of the EME spec provided by the Google Clear Key tests. DONE

2.           Convert current tests to multi-DRM: Modify the Clear Key tests so that they can be used in the CableLabs defined multi-DRM environment. BEGUN

3.           Fill in the gaps in coverage: Extend the existing test framework to cover all tests the editors and W3C consider to be required. UNDER DISCUSSION



Details:

Coverage Analysis:

An analysis was completed of the existing Clear Key based EME tests, mapping them to the EME specification. It was determined that converting the existing clear key tests gives us moderate coverage. Here is a link to that analysis:



-            https://rawgit.com/jdsmith3000/encrypted-media-testcoverage/master/index.html




Modifying Google Clear Key Tests:

Google submitted a set of EME unit tests to W3C (https://github.com/w3c/web-platform-tests/tree/master/encrypted-media/Google). The tests cover Clear Key, but are planned to be extended to cover multi-DRM. Each test needs to be extended for each supported DRM and multi-DRM signaling must be included. The updated tests should then detect and test each CDM supported by the user agent.



As Greg Rutz made clear in his previous posting, we plan to update existing Clear Key tests to use existing multi-purpose DRM-license request logic from CastLabs.  This process has begun, converting a small number of selected Clear Key tests to multi-DRM.  Once that is completed, we will develop a generalized approach to multi-Format and multi-DRM testing for the remaining tests.  We hope to distribute additional test conversion work across other resources.



Filling in the Gaps in Coverage

The test coverage report identifies EME spec methods and attributes which are not covered by a test plan based on the existing Clear Key tests. Identified gaps include:



-            3.1.2.2 Get Supported Configuration and Consent

-            3.1.2.3 Get Supported Capabilities for Audio/Video Type

-            3.1.2.5 Get Consent Status

-            3.2 MediaKeySystemConfiguration: distinctiveIdentifier of type MediaKeysRequirement, defaulting to "optional"

-            3.2 MediaKeySystemConfiguration: label of type DOMString, defaulting to ""

-            3.2 MediaKeySystemConfiguration: persistentState of type MediaKeysRequirement, defaulting to "optional"

-            3.2 MediaKeySystemConfiguration: sessionTypes of type sequence

-            5.2.2 CDM Unavailable

-            5.3 Storage and Persistence

-            6.1 MediaKeySession Attributes: expiration of type unrestricted double, readonly

-            6.1 MediaKeySession Attributes: onkeystatuseschange of type EventHandler

-            6.2 MediaKeySession Methods: load

-            6.6.2 Update Key Statuses

-            6.6.3 Update Expiration

-            6.6.5 MediaKeySession Destroyed

-            6.6.6 Monitor for CDM State Changes

-            6.8 Session Storage and Persistence

-            7.5.5 Attempt to Resume Playback If Necessary

-            8.3 Support Multiple Keys



These are a small minority (19 out of 136 requirements analyzed).  It has not yet been decided which of these gaps need to be addressed in the final test plan.



-----Original Message-----
From: Francois Daoust [mailto:fd@w3.org<mailto:fd@w3.org>]
Sent: Monday, July 18, 2016 6:16 AM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>
Cc: David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> (ddorwin@google.com<mailto:ddorwin@google.com>) <ddorwin@google.com<mailto:ddorwin@google.com>>; Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>
Subject: Re: [Agenda] HTML Media Extensions WG Editors meeting, Tue Jul 19



Hi all,



I cannot attend the call tomorrow, and will not be around in the next couple of weeks. A quick update below:



On 15/07/2016 04:39, Paul Cotton wrote:

> Thanks for the update, Matt.

>

> Could others provide a short written summary of what you are working on and what blockers you think we have or where you need others to help?



Regarding MSE testing, as shared on the HTML Media mailing-list, things are starting to look good. See that email for a list of main missing features in implementations:

https://lists.w3.org/Archives/Public/public-html-media/2016Jul/0006.html




I corrected tests that failed because they incorrectly truncated the duration of buffered content, which is no longer allowed. This turned a number of cells green. I also imported and converted tests from the Chromium test suite.



New/Corrected MSE tests have not been merged in the Web Platform Tests repository yet. They should first be reviewed by someone else (I see Matt is planning to do so) to ensure I got things right. List at:

https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aopen+label%3Amedia-source




Some edge cases of SourceBuffer algorithms are probably not thoroughly tested. That said, the test suite covers main visible effects of these algorithms, which I hope is good enough.



In short:

1. Outstanding PR on test repo should be reviewed, fixed as needed, and merged.

2. I think the test suite can be viewed as "good enough" after that, although it can certainly be improved.

3. Some features are not yet implemented across browsers. The implementation report should be updated once that is done. I'd be happy to do that once I am back to work.



Thanks,

Francois.







>

> Sent from my Windows 10 phone

>

> *From: *Matt Wolenetz <mailto:wolenetz@google.com>

> *Sent: *July 14, 2016 14:58

> *To: *Paul Cotton <mailto:Paul.Cotton@microsoft.com>

> *Cc: *David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> (ddorwin@google.com<mailto:ddorwin@google.com>) <mailto:ddorwin@google.com>; Mark Watson <mailto:watsonm@netflix.com>; Jerry Smith (WPT) <mailto:jdsmith@microsoft.com>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <mailto:plh@w3.org>; Francois Daoust <mailto:fd@w3.org>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org> <mailto:public-hme-editors@w3.org>; Iraj Sodagar <mailto:irajs@microsoft.com>; John Simmons <mailto:johnsim@microsoft.com>

> *Subject: *Re: [Agenda] HTML Media Extensions WG Editors meeting, Tue Jul 19

>

> Acknowledged. I'll attend.

>

> Regarding: https://github.com/w3c/media-source/milestones/V1Editorial, I currently have 9 pull requests out for review at https://github.com/w3c/media-source/pulls/wolenetz and am working on more of those.

> I'm also planning to review outstanding w-p-t PRs and upstreaming more tests from Chromium prior to the meeting, in parallel with closing down more Chromium MSE spec compatibility issues discovered during testing.

>

> Thanks,

> Matt

>

>

> On Thu, Jul 14, 2016 at 2:40 PM, Paul Cotton <Paul.Cotton@microsoft.com <mailto:Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com%20%3cmailto:Paul.Cotton@microsoft.com>>> wrote:

>

>     I am scheduling another meeting of the Media Extensions WG MSE and EME Editors for Tuesday Jul 19 at 8am PDT during the normal WG/TF meeting slot.  Meeting location information is given below.____

>

>     __ __

>

>     Please let me know if there are any others I should invite to this meeting.____

>

>     __ __

>

>     __1.__MSE and EME timeline discussion, Paul and Philippe

>     https://lists.w3.org/Archives/Public/public-html-media/2016May/0029.html ____

>

>     Status:  The timeline indicates we are planning to do the CfC for MSE and EME Proposed Recommendations on Aug 2. ____

>

>     __2.__MSE test suite and testing report status, Francois Daoust

>     https://lists.w3.org/Archives/Public/public-html-media/2016Jul/0006.html


>     http://w3c.github.io/test-results/media-source/complete-fails.html ____

>

>     __3.__EME test suite

>     https://lists.w3.org/Archives/Public/public-html-media/2016Jul/0004.html ____

>

>     __4.__Schedule for completion of editorial MSE and EME issues

>     https://github.com/w3c/media-source/milestones/V1Editorial ____

>

>     https://github.com/w3c/encrypted-media/milestones/V1Editorial ____

>

>     __5.__MSE Registry publication as WG notes, Philippe____

>

>     https://github.com/w3c/media-source/issues/74 ____

>

>     __6.__Any other business____

>

>     __ __

>

>     /paulc____

>

>     __ __

>

>     Paul Cotton, Microsoft Ca - nada____

>

>     17 Eleanor Drive, Ottawa, Ontario K2E 6A3____

>

>     Tel: (425) 705-9596<tel:%28425%29%20705-9596> <tel:%28425%29%20705-9596> Fax: (425) 936-7329<tel:%28425%29%20936-7329> <tel:%28425%29%20936-7329>____

>

>     __ __

>

>     Meeting information:____

>

>     http://irc.w3.org #html-media____

>

>     __ __

>

>     Join WebEx meeting____

>

>     Meeting number:           649 602 452 Meeting password:        media ____

>

>     ____

>

>       Join by phone ____

>

>     +1-617-324-0000<tel:%2B1-617-324-0000> <tel:%2B1-617-324-0000> US Toll Number____

>

>     Access code: 649 602 452 ____

>

>     __ __

>

>     __ __

>

>     __ __

>

>