Re: EME Test Status Update - 10-27-16

The failure in /encrypted-media/drm-mp4-playback-temporary-events.html (at
least in Chrome and in Firefox) is caused by Widevine CDM resolving
MediaKeySession.closed and the promise returned by MediaKeySession.close()
in the wrong order.

The test changed with issue 4058
<https://github.com/w3c/web-platform-tests/issues/4058> to expect that the
MediaKeySession.closed promise is resolved before the promise returned by
MediaKeySession.close() is resolved. Both Firefox and Chrome depend on the
Widevine CDM to resolve the promise in the correct order, and it doesn't.
So this is an implementation bug in the Widevine CDM.

The ClearKey variant of this test
<https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
should be fixed in Firefox Nightly in a day or so, and is already fixed in
the latest Chrome Canary.

We have patches in the works for fixing
drm-mp4-playback-temporary-multikey-sequential[-readyState].html.

Chris.



On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> Summary of issues related to each failure is:
>
>
>
> Persistent-license failures - 1 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
>
>
>      This is being internally investigated at Google for a Chrome OS fix.
>
>
>
> Temporary session failures - 2 Failures
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
>
>
>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
> assigned to mwatson2
>
>
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, playback, check events
>
>
>
>      This is actually a new failure that didn’t occur on the 10/24/2016
> pass.  Can anyone comment on the test?
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
>
>
>      This issue is dependent on some spec clarifications in these issues:
>
>      https://github.com/w3c/encrypted-media/issues/338 - closed
>
>      https://github.com/w3c/encrypted-media/issues/339 - closed
>
>      https://github.com/w3c/encrypted-media/issues/336 - in review
>
>
>
>      I’m not sure if we have an issue opened to modify the test once the
> spec issues are cleared.
>
>
>
> Jerry
>
>
>
> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
> *Sent:* Thursday, October 27, 2016 4:40 PM
> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
> wolenetz@google.com>; 'public-html-media@w3.org' <public-html-media@w3.org
> >
> *Subject:* EME Test Status Update - 10-27-16
>
>
>
> I’ve posted updated test status reports to reflect a few bug fixes and
> also to add Chrome OS results.
>
>
>
> Reports:
>
>
>
> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>
> o   *Test files: 97; Total subtests: 306*
>
> ·        http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> o   *Completely failed files: 16; Completely failed subtests: 16; Failure
> level: 16/306 (5.23%)*
>
> ·        http://w3c.github.io/test-results/encrypted-media/less-
> than-2.html
>
> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
> Failure level: 40/306 (13.07%)*
>
> Comments:
>
>
>
> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from 47
> to 40 subtest failures.
>
> -   Persistent-license failures dropped from 5 to 1 with the addition of
> Chrome on Chrome OS results.
>
> -   One temporary session test failed that was previously passing:
>
> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
> 100.00%, 0.33% of total)
>
> o   drm, temporary, mp4, playback, check events
>
> -   There are only 5 tests in complete-fails that we are still working on
> for our conformance record.
>
> -   That reflects our revised criteria of having a single passing
> implementation (in addition to broad interoperable experience).
>
> In line with that criteria, I’ve chosen to break down just the
> complete-fails test results for this summary.
>
>
>
> Browser Versions:
>
>
>
> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>
> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>
> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>
> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>
>
>
> Summary Groupings;
>
>
>
> Errors (considered bugs, not blocking)
> 0
>
> Clear Key persistent-license (not planned, may make non-normative)
> 5
>
> Clear Key persistent-usage-record (at risk, planned to be removed)
> 3
>
> Clear Key transitions between clear and encrypted (bugs, not blocking)
> 0
>
>
>
> Not Blocking Tests
>  8
>
>
>
> Persistent-license (fixes in work)
>           1
>
> Persistent-usage-record (at risk, planned to be removed)
> 3
>
> Temporary session edge cases (bugs, not block, fixes in work)
> 2
>
> Temporary session readyState (issues under discussion, at risk)
>  2
>
>
>
> In work or at risk
> 8
>
>
>
> TOTAL
> 16
>
>
>
>
>
> Details on failed tests are below.
>
>
>
> Jerry
>
>
>
>
>
> Error handling (considered bugs, not blocking) - 0 Failures
>
>
>
> Clear key persistent-license tests (not planned to be implemented, may
> make informative note) - 5 Failures
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>
>
>
> Clear key persistent-usage-record (not planned to be implemented) - 3
> Failures
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
> window
>
>
>
> Clear key transitions between clear and encrypted content (considered
> bugs, not blocking) - 2 Failures
>
>
>
> Persistent-license failures - 6 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
>
>
> Persistent-usage-record failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4playback
>
>
>
> Temporary session failures - 2 Failures
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, playback, check events
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>