- From: Paul Cotton <Paul.Cotton@microsoft.com>
- Date: Tue, 18 Oct 2016 15:10:01 +0000
- To: Chris Pearce <cpearce@mozilla.com>, "Jerry Smith (WPT)" <jdsmith@microsoft.com>
- CC: David Dorwin <ddorwin@google.com>, Matt Wolenetz <wolenetz@google.com>, Mark Watson <watsonm@netflix.com>, "public-html-media@w3.org" <public-html-media@w3.org>
- Message-ID: <CY4PR03MB2728E64492EBBDBF0807D844EAD30@CY4PR03MB2728.namprd03.prod.outlook.com>
Could you let Jerry know when he should re-test the Mozilla and which fixes have landed and which are pending?
/paulc
From: Chris Pearce [mailto:cpearce@mozilla.com]
Sent: Monday, October 17, 2016 8:41 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <ddorwin@google.com>; Matt Wolenetz <wolenetz@google.com>; Mark Watson <watsonm@netflix.com>; public-html-media@w3.org
Subject: Re: EME test status
Mozilla is working on making the EME Web Platform Tests in the "Wrong Error Handling", "Temporary Session", and "Misc" categories pass in Firefox 52, which is in our Nightly release channel. We have submitted several pull requests for bugs in the tests, fixed a number of issues in Firefox, and the number of tests that pass in these categories in Firefox Nightly is increasing daily.
At this time we are not planning on implementing "persistent-usage-record" or "persistent-license" sessions. We won't be trying to make the "Persistent-License Sessions" and "Persistent-Usage-Record" tests pass in the EME v1 time frame.
Cheers,
Chris Pearce.
On Thu, Oct 13, 2016 at 1:32 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
There were some minor bug fixes in the text suite today. I’ve run the tests and posted reports:
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html
o Test files: 97; Total subtests: 306
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/less-than-2.html
o Test files without 2 passes: 37; Subtests without 2 passes: 68; Failure level: 68/306 (22.22%)
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/complete-fails.html
o Completely failed files: undefined; Completely failed subtests: 44; Failure level: 44/306 (14.38%)
Our test suite work is scheduled to be done on October 25th, so we have just under 2 weeks remaining. Given the limited time remaining, I recommend we focus on closures and not on adding any new tests. I’ve grouped the less than 2 results below to help draw some conclusions using these groupings:
- Wrong Error Handling: 42 failures are this type (out of 68 total failures). It would be great to do an implementation push to get these passing, since conceptually it would leave only 26 remaining failures/timeouts. The majority of these are returning the wrong error.
- Temporary Session: 6 of these are failing and I believe some are already being investigated. Can they be closed by next week?
- Persistent-License Sessions: 11 (all tests) are failing, which suggests that Persistent-License should be considered “at risk”. A few of these pass with Polyfills.
- Persistent-Usage-Record: 6 (all tests) are failing. This feature is already listed at risk in the current CR.
- Misc: 3 tests don’t really map to any of the above.
It would be good to confirm:
- Whether Chrome and Firefox can make fixes to close the Error handling failures.
- What the status is on the 6 Temporary session bug investigations.
- Whether any work is planned on Persistent-License failures.
- Whether any work is planned on Persistent-Usage-Record failures.
Test groupings are below.
Jerry
Test
CH56
ED14
FF52
Error Handling: These failures are a result of either returning the wrong error type, or not properly failing a promise after an error. Total failures: 42
/encrypted-media/clearkey-generate-request-disallowed-input.html (7/7, 100.00%, 2.29% of total)
OK
TIMEOUT
OK
org.w3.clearkey, temporary, webm, initData longer than 64Kb characters
FAIL
-
FAIL
org.w3.clearkey, temporary, cenc, initData longer than 64Kb characters
FAIL
-
FAIL
org.w3.clearkey, temporary, keyids, initData longer than 64Kb characters
FAIL
-
FAIL
org.w3.clearkey, temporary, cenc, invalid initdata (invalid pssh)
FAIL
-
FAIL
org.w3.clearkey, temporary, cenc, invalid initdata (not pssh)
FAIL
-
FAIL
org.w3.clearkey, temporary, keyids, invalid initdata (too short key ID)
FAIL
-
FAIL
org.w3.clearkey, temporary, keyids, invalid initdata (too long key ID)
FAIL
-
FAIL
/encrypted-media/clearkey-mp4-syntax-mediakeysession.html (4/7, 57.14%, 1.31% of total)
OK
OK
OK
org.w3.clearkey test MediaKeySession generateRequest() exceptions.
PASS
FAIL
FAIL
org.w3.clearkey test MediaKeySession load() exceptions.
FAIL
FAIL
FAIL
org.w3.clearkey test if MediaKeySession generateRequest() resolves for various sessions
PASS
FAIL
FAIL
org.w3.clearkey test MediaKeySession update() exceptions.
PASS
FAIL
FAIL
/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (3/58, 5.17%, 0.98% of total)
OK
OK
OK
org.w3.clearkey, requestMediaKeySystemAccess: Empty Key System should result in TypeError
FAIL
FAIL
FAIL
org.w3.clearkey, requestMediaKeySystemAccess: Empty supportedConfigurations should result in TypeError
FAIL
FAIL
FAIL
Trailing space in contentType
FAIL
FAIL
PASS
/encrypted-media/clearkey-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33% of total)
OK
OK
OK
org.w3.clearkey test MediaKeys setServerCertificate() exceptions.
FAIL
FAIL
PASS
/encrypted-media/clearkey-mp4-syntax-mediakeysystemaccess.html (1/2, 50.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey test requestMediaKeySystemAccess() exceptions.
FAIL
FAIL
FAIL
/encrypted-media/drm-generate-request-disallowed-input.html (7/7, 100.00%, 2.29% of total)
OK
OK
OK
drm, temporary, webm, initData longer than 64Kb characters
FAIL
-
FAIL
drm, temporary, cenc, initData longer than 64Kb characters
FAIL
FAIL
FAIL
drm, temporary, cenc, invalid initdata (invalid pssh)
FAIL
FAIL
FAIL
drm, temporary, cenc, invalid initdata (not pssh)
FAIL
FAIL
FAIL
drm, temporary, keyids, initData longer than 64Kb characters
-
FAIL
FAIL
drm, temporary, keyids, invalid initdata (too short key ID)
-
FAIL
FAIL
drm, temporary, keyids, invalid initdata (too long key ID)
-
FAIL
FAIL
/encrypted-media/drm-mp4-syntax-mediakeysession.html (3/7, 42.86%, 0.98% of total)
OK
OK
OK
drm test MediaKeySession generateRequest() exceptions.
PASS
FAIL
FAIL
drm test MediaKeySession load() exceptions.
FAIL
FAIL
FAIL
drm test MediaKeySession update() exceptions.
PASS
FAIL
FAIL
/encrypted-media/drm-mp4-requestmediakeysystemaccess.html (4/58, 6.90%, 1.31% of total)
OK
OK
OK
drm, requestMediaKeySystemAccess: Empty Key System should result in TypeError
FAIL
FAIL
FAIL
drm, requestMediaKeySystemAccess: Empty supportedConfigurations should result in TypeError
FAIL
FAIL
FAIL
drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4; foo="bar"') should result in NotSupportedError
FAIL
FAIL
PASS
drm, requestMediaKeySystemAccess: Unrecognized foo with codecs ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in NotSupportedError
FAIL
FAIL
PASS
/encrypted-media/drm-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33% of total)
OK
OK
OK
drm test MediaKeys setServerCertificate() exceptions.
FAIL
FAIL
PASS
/encrypted-media/drm-mp4-syntax-mediakeysystemaccess.html (1/2, 50.00%, 0.33% of total)
OK
OK
OK
drm test requestMediaKeySystemAccess() exceptions.
FAIL
FAIL
FAIL
/encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
OK
OK
OK
Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object Object])
FAIL
FAIL
PASS
HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
FAIL
FAIL
PASS
MediaKeySystemAccess interface: operation createMediaKeys()
FAIL
FAIL
PASS
MediaKeys interface: operation setServerCertificate(BufferSource)
FAIL
FAIL
PASS
MediaKeySession interface: operation generateRequest(DOMString,BufferSource)
FAIL
FAIL
PASS
MediaKeySession interface: operation load(DOMString)
FAIL
FAIL
PASS
MediaKeySession interface: operation update(BufferSource)
FAIL
FAIL
PASS
MediaKeySession interface: operation close()
FAIL
FAIL
PASS
MediaKeySession interface: operation remove()
FAIL
FAIL
PASS
MediaKeyMessageEvent interface object length
FAIL
FAIL
FAIL
Temporary Sessions: These failures are in tests that evaluate specific protected content scenarios. Many temporary sessions are supported today, but not these variants. Total failures: 6
/encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, temporary, mp4, playback, single key, clear then encrypted content
TIMEOUT
TIMEOUT
PASS
/encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, temporary, mp4, playback, single key, encrypted then clear content
TIMEOUT
TIMEOUT
PASS
/encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, temporary, mp4, expired license
FAIL
FAIL
FAIL
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential, readyState
FAIL
TIMEOUT
FAIL
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
OK
TIMEOUT
OK
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
FAIL
-
FAIL
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
OK
TIMEOUT
OK
drm, successful playback, temporary, mp4, multiple keys, sequential
FAIL
-
FAIL
Persistent-License Sessions: These failures are in tests that evaluate persistent license scenarios. These are largely unsupported today. Total failures: 11
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
POLYFILL
TIMEOUT
POLYFILL
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-license, mp4, playback, check events
POLYFILL
TIMEOUT
POLYFILL
/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-license, mp4playback
POLYFILL
TIMEOUT
POLYFILL
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, cannot load persistent license into temporary session
FAIL
FAIL
FAIL
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-license, mp4, playback, destroy and acknowledge
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-license, mp4, playback, check events
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-license, mp4playback
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-license, mp4, playback, retrieve, playback and destroy
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-license, mp4, playback, retrieve, playback
TIMEOUT
TIMEOUT
TIMEOUT
Persistent-Usage-Record Sessions: These failures are in tests that evaluate persistent usage record scenarios. These are largely unsupported today. Total failures: 6
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-usage-record, mp4, playback, check events
POLYFILL
TIMEOUT
POLYFILL
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-usage-record, mp4playback
POLYFILL
TIMEOUT
POLYFILL
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window
TIMEOUT
TIMEOUT
TIMEOUT
/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-usage-record, mp4playback
TIMEOUT
FAIL
TIMEOUT
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-usage-record, mp4, playback, check events
TIMEOUT
FAIL
TIMEOUT
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, persistent-usage-record, mp4, playback, retrieve in new window
TIMEOUT
FAIL
TIMEOUT
Misc: Multiple key scenarios and one stress case. Minor impact on proving out the spec. Total failures: 3
/encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-different-mediakeys.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
org.w3.clearkey, setmediakeys multiple times with different mediakeys
PASS
FAIL
FAIL
/encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
update() with invalid response (longer than 64Kb characters) should fail.
PASS
FAIL
FAIL
/encrypted-media/drm-mp4-setmediakeys-multiple-times-with-different-mediakeys.html (1/1, 100.00%, 0.33% of total)
OK
OK
OK
drm, setmediakeys multiple times with different mediakeys
PASS
FAIL
FAIL
Received on Tuesday, 18 October 2016 15:10:41 UTC