- From: Jerry Smith (WPT) <jdsmith@microsoft.com>
- Date: Mon, 31 Oct 2016 23:24:55 +0000
- To: David Dorwin <ddorwin@google.com>, Chris Pearce <cpearce@mozilla.com>
- CC: Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "public-html-media@w3.org" <public-html-media@w3.org>
- Message-ID: <BY2PR03MB041C6F4F8DCD2AEA58A8ABDA4AE0@BY2PR03MB041.namprd03.prod.outlook.com>
We seem very close to finishing. Do we have a shot at closing open test issues this week?
David; Can you clarify if you consider the new issues you’ve listed as V1? I think you’ve added these two to the list:
- drm-mp4-playback-temporary-expired.html
o Flaky due to timing.
- drm-mp4-unique-origin.html
o New failure
My previous list:
- drm-mp4-playback-persistent-license-events.html
o https://crbug.com/660393
o ETA?
- drm-temporary-license-type.html
Widevine CDM returns a temporary license when a persistent one is requested.
Proposal in https://github.com/w3c/web-platform-tests/issues/4027 to rework the test.
Is this must fix for V1?
- drm-mp4-playback-temporary-events.html
o https://crbug.com/659828
o ETA?
- drm-mp4-playback-temporary-multikey-sequential-readyState.html
- drm-mp4-playback-temporary-multikey-sequential.html
o Am I correct that multiple PRs are targeted at this issue?
o Chris Pearce reports fixes are in work.
o What’s the prospects for closing this week?
I added a V1 milestone to the test repository and marked one issue V1 (https://github.com/w3c/web-platform-tests/issues/4027) as V1. There are 10 other test issues without a milestone, and 5 marked VNext. Can folks weigh in on which should be treated as V1?
Jerry
From: David Dorwin [mailto:ddorwin@google.com]
Sent: Friday, October 28, 2016 9:10 AM
To: Chris Pearce <cpearce@mozilla.com>
Cc: Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; public-html-media@w3.org
Subject: Re: EME Test Status Update - 10-27-16
Thanks, Jerry.
The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html failure is tracked in https://crbug.com/659828.
The drm-mp4-playback-persistent-license-events.html failure on Chrome OS is tracked as https://crbug.com/660393.
drm-mp4-playback-temporary-expired.html is flaky due to timing. I just saw a failure and a pass.
drm-mp4-unique-origin.html appears to be a new failure. We're looking into it.
On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>> wrote:
The failure in /encrypted-media/drm-mp4-playback-temporary-events.html (at least in Chrome and in Firefox) is caused by Widevine CDM resolving MediaKeySession.closed and the promise returned by MediaKeySession.close() in the wrong order.
The test changed with issue 4058<https://github.com/w3c/web-platform-tests/issues/4058> to expect that the MediaKeySession.closed promise is resolved before the promise returned by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on the Widevine CDM to resolve the promise in the correct order, and it doesn't. So this is an implementation bug in the Widevine CDM.
The ClearKey variant of this test<https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html> should be fixed in Firefox Nightly in a day or so, and is already fixed in the latest Chrome Canary.
We have patches in the works for fixing drm-mp4-playback-temporary-multikey-sequential[-readyState].html.
Chris.
On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
Summary of issues related to each failure is:
Persistent-license failures - 1 Failures
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events
This is being internally investigated at Google for a Chrome OS fix.
Temporary session failures - 2 Failures
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
This is being worked under https://github.com/w3c/web-platform-tests/issues/4027 assigned to mwatson2<https://github.com/w3c/web-platform-tests/issues/4027%20assigned%20to%20mwatson2>
/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, playback, check events
This is actually a new failure that didn’t occur on the 10/24/2016 pass. Can anyone comment on the test?
Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential
This issue is dependent on some spec clarifications in these issues:
https://github.com/w3c/encrypted-media/issues/338 - closed
https://github.com/w3c/encrypted-media/issues/339 - closed
https://github.com/w3c/encrypted-media/issues/336 - in review
I’m not sure if we have an issue opened to modify the test once the spec issues are cleared.
Jerry
From: Jerry Smith (WPT) [mailto:jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>]
Sent: Thursday, October 27, 2016 4:40 PM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>; Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; 'public-html-media@w3.org<mailto:public-html-media@w3.org>' <public-html-media@w3.org<mailto:public-html-media@w3.org>>
Subject: EME Test Status Update - 10-27-16
I’ve posted updated test status reports to reflect a few bug fixes and also to add Chrome OS results.
Reports:
• http://w3c.github.io/test-results/encrypted-media/all.html
o Test files: 97; Total subtests: 306
• http://w3c.github.io/test-results/encrypted-media/complete-fails.html
o Completely failed files: 16; Completely failed subtests: 16; Failure level: 16/306 (5.23%)
• http://w3c.github.io/test-results/encrypted-media/less-than-2.html
o Test files without 2 passes: 28; Subtests without 2 passes: 40; Failure level: 40/306 (13.07%)
Comments:
- Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from 47 to 40 subtest failures.
- Persistent-license failures dropped from 5 to 1 with the addition of Chrome on Chrome OS results.
- One temporary session test failed that was previously passing:
o /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
o drm, temporary, mp4, playback, check events
- There are only 5 tests in complete-fails that we are still working on for our conformance record.
- That reflects our revised criteria of having a single passing implementation (in addition to broad interoperable experience).
In line with that criteria, I’ve chosen to break down just the complete-fails test results for this summary.
Browser Versions:
CD56: Chrome 56.0.2902.0 canary (64-bit)
CO56: Chrome OS 56.0.2900.0 canary (64-bit)
ED14: Edge 14, Windows 10.0.14393.351 (64-bit)
FF52: Firefox 52.0a1 (2016-10-27) (32-bit)
Summary Groupings;
Errors (considered bugs, not blocking) 0
Clear Key persistent-license (not planned, may make non-normative) 5
Clear Key persistent-usage-record (at risk, planned to be removed) 3
Clear Key transitions between clear and encrypted (bugs, not blocking) 0
Not Blocking Tests 8
Persistent-license (fixes in work) 1
Persistent-usage-record (at risk, planned to be removed) 3
Temporary session edge cases (bugs, not block, fixes in work) 2
Temporary session readyState (issues under discussion, at risk) 2
In work or at risk 8
TOTAL 16
Details on failed tests are below.
Jerry
Error handling (considered bugs, not blocking) - 0 Failures
Clear key persistent-license tests (not planned to be implemented, may make informative note) - 5 Failures
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-license.ht<http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
Clear key persistent-usage-record (not planned to be implemented) - 3 Failures
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window
Clear key transitions between clear and encrypted content (considered bugs, not blocking) - 2 Failures
Persistent-license failures - 6 Failures
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events
Persistent-usage-record failures - 3 Failures
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, check events
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, retrieve in new window
/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4playback
Temporary session failures - 2 Failures
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, playback, check events
Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential
Received on Monday, 31 October 2016 23:26:16 UTC