EME test status

There were some minor bug fixes in the text suite today.  I’ve run the tests and posted reports:


-                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html


o   Test files: 97; Total subtests: 306

-                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/less-than-2.html


o   Test files without 2 passes: 37; Subtests without 2 passes: 68; Failure level: 68/306 (22.22%)

-                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/complete-fails.html


o   Completely failed files: undefined; Completely failed subtests: 44; Failure level: 44/306 (14.38%)

Our test suite work is scheduled to be done on October 25th, so we have just under 2 weeks remaining.  Given the limited time remaining, I recommend we focus on closures and not on adding any new tests.  I’ve grouped the less than 2 results below to help draw some conclusions using these groupings:


-          Wrong Error Handling:                 42 failures are this type (out of 68 total failures).  It would be great to do an implementation push to get these passing, since conceptually it would leave only 26 remaining failures/timeouts.  The majority of these are returning the wrong error.

-          Temporary Session:                      6 of these are failing and I believe some are already being investigated.  Can they be closed by next week?

-          Persistent-License Sessions:        11 (all tests) are failing, which suggests that Persistent-License should be considered “at risk”.  A few of these pass with Polyfills.

-          Persistent-Usage-Record:            6 (all tests) are failing.  This feature is already listed at risk in the current CR.

-          Misc:                                              3 tests don’t really map to any of the above.

It would be good to confirm:


-          Whether Chrome and Firefox can make fixes to close the Error handling failures.

-          What the status is on the 6 Temporary session bug investigations.

-          Whether any work is planned on Persistent-License failures.

-          Whether any work is planned on Persistent-Usage-Record failures.

Test groupings are below.

Jerry

Test

CH56

ED14

FF52


Error Handling:  These failures are a result of either returning the wrong error type, or not properly failing a promise after an error.  Total failures:  42


/encrypted-media/clearkey-generate-request-disallowed-input.html (7/7, 100.00%, 2.29% of total)

OK

TIMEOUT

OK

org.w3.clearkey, temporary, webm, initData longer than 64Kb characters

FAIL

-

FAIL

org.w3.clearkey, temporary, cenc, initData longer than 64Kb characters

FAIL

-

FAIL

org.w3.clearkey, temporary, keyids, initData longer than 64Kb characters

FAIL

-

FAIL

org.w3.clearkey, temporary, cenc, invalid initdata (invalid pssh)

FAIL

-

FAIL

org.w3.clearkey, temporary, cenc, invalid initdata (not pssh)

FAIL

-

FAIL

org.w3.clearkey, temporary, keyids, invalid initdata (too short key ID)

FAIL

-

FAIL

org.w3.clearkey, temporary, keyids, invalid initdata (too long key ID)

FAIL

-

FAIL

/encrypted-media/clearkey-mp4-syntax-mediakeysession.html (4/7, 57.14%, 1.31% of total)

OK

OK

OK

org.w3.clearkey test MediaKeySession generateRequest() exceptions.

PASS

FAIL

FAIL

org.w3.clearkey test MediaKeySession load() exceptions.

FAIL

FAIL

FAIL

org.w3.clearkey test if MediaKeySession generateRequest() resolves for various sessions

PASS

FAIL

FAIL

org.w3.clearkey test MediaKeySession update() exceptions.

PASS

FAIL

FAIL

/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (3/58, 5.17%, 0.98% of total)

OK

OK

OK

org.w3.clearkey, requestMediaKeySystemAccess: Empty Key System should result in TypeError

FAIL

FAIL

FAIL

org.w3.clearkey, requestMediaKeySystemAccess: Empty supportedConfigurations should result in TypeError

FAIL

FAIL

FAIL

Trailing space in contentType

FAIL

FAIL

PASS

/encrypted-media/clearkey-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33% of total)

OK

OK

OK

org.w3.clearkey test MediaKeys setServerCertificate() exceptions.

FAIL

FAIL

PASS

/encrypted-media/clearkey-mp4-syntax-mediakeysystemaccess.html (1/2, 50.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey test requestMediaKeySystemAccess() exceptions.

FAIL

FAIL

FAIL

/encrypted-media/drm-generate-request-disallowed-input.html (7/7, 100.00%, 2.29% of total)

OK

OK

OK

drm, temporary, webm, initData longer than 64Kb characters

FAIL

-

FAIL

drm, temporary, cenc, initData longer than 64Kb characters

FAIL

FAIL

FAIL

drm, temporary, cenc, invalid initdata (invalid pssh)

FAIL

FAIL

FAIL

drm, temporary, cenc, invalid initdata (not pssh)

FAIL

FAIL

FAIL

drm, temporary, keyids, initData longer than 64Kb characters

-

FAIL

FAIL

drm, temporary, keyids, invalid initdata (too short key ID)

-

FAIL

FAIL

drm, temporary, keyids, invalid initdata (too long key ID)

-

FAIL

FAIL

/encrypted-media/drm-mp4-syntax-mediakeysession.html (3/7, 42.86%, 0.98% of total)

OK

OK

OK

drm test MediaKeySession generateRequest() exceptions.

PASS

FAIL

FAIL

drm test MediaKeySession load() exceptions.

FAIL

FAIL

FAIL

drm test MediaKeySession update() exceptions.

PASS

FAIL

FAIL

/encrypted-media/drm-mp4-requestmediakeysystemaccess.html (4/58, 6.90%, 1.31% of total)

OK

OK

OK

drm, requestMediaKeySystemAccess: Empty Key System should result in TypeError

FAIL

FAIL

FAIL

drm, requestMediaKeySystemAccess: Empty supportedConfigurations should result in TypeError

FAIL

FAIL

FAIL

drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4; foo="bar"') should result in NotSupportedError

FAIL

FAIL

PASS

drm, requestMediaKeySystemAccess: Unrecognized foo with codecs ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in NotSupportedError

FAIL

FAIL

PASS

/encrypted-media/drm-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33% of total)

OK

OK

OK

drm test MediaKeys setServerCertificate() exceptions.

FAIL

FAIL

PASS

/encrypted-media/drm-mp4-syntax-mediakeysystemaccess.html (1/2, 50.00%, 0.33% of total)

OK

OK

OK

drm test requestMediaKeySystemAccess() exceptions.

FAIL

FAIL

FAIL

/encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)

OK

OK

OK

Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object Object])

FAIL

FAIL

PASS

HTMLMediaElement interface: operation setMediaKeys(MediaKeys)

FAIL

FAIL

PASS

MediaKeySystemAccess interface: operation createMediaKeys()

FAIL

FAIL

PASS

MediaKeys interface: operation setServerCertificate(BufferSource)

FAIL

FAIL

PASS

MediaKeySession interface: operation generateRequest(DOMString,BufferSource)

FAIL

FAIL

PASS

MediaKeySession interface: operation load(DOMString)

FAIL

FAIL

PASS

MediaKeySession interface: operation update(BufferSource)

FAIL

FAIL

PASS

MediaKeySession interface: operation close()

FAIL

FAIL

PASS

MediaKeySession interface: operation remove()

FAIL

FAIL

PASS

MediaKeyMessageEvent interface object length

FAIL

FAIL

FAIL


Temporary Sessions:  These failures are in tests that evaluate specific protected content scenarios.  Many temporary sessions are supported today, but not these variants.  Total failures: 6


/encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, temporary, mp4, playback, single key, clear then encrypted content

TIMEOUT

TIMEOUT

PASS

/encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, temporary, mp4, playback, single key, encrypted then clear content

TIMEOUT

TIMEOUT

PASS

/encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, temporary, mp4, expired license

FAIL

FAIL

FAIL

/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential, readyState

FAIL

TIMEOUT

FAIL

/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)

OK

TIMEOUT

OK

drm, successful playback, temporary, mp4, multiple keys, sequential, readyState

FAIL

-

FAIL

/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)

OK

TIMEOUT

OK

drm, successful playback, temporary, mp4, multiple keys, sequential

FAIL

-

FAIL


Persistent-License Sessions:  These failures are in tests that evaluate persistent license scenarios.  These are largely unsupported today.  Total failures: 11


/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge

POLYFILL

TIMEOUT

POLYFILL

/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-license, mp4, playback, check events

POLYFILL

TIMEOUT

POLYFILL

/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-license, mp4playback

POLYFILL

TIMEOUT

POLYFILL

/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, cannot load persistent license into temporary session

FAIL

FAIL

FAIL

/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-license, mp4, playback, destroy and acknowledge

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-license, mp4, playback, check events

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-license, mp4playback

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-license, mp4, playback, retrieve, playback and destroy

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-license, mp4, playback, retrieve, playback

TIMEOUT

TIMEOUT

TIMEOUT


Persistent-Usage-Record Sessions:  These failures are in tests that evaluate persistent usage record scenarios.  These are largely unsupported today.  Total failures: 6


/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-usage-record, mp4, playback, check events

POLYFILL

TIMEOUT

POLYFILL

/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-usage-record, mp4playback

POLYFILL

TIMEOUT

POLYFILL

/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window

TIMEOUT

TIMEOUT

TIMEOUT

/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-usage-record, mp4playback

TIMEOUT

FAIL

TIMEOUT

/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-usage-record, mp4, playback, check events

TIMEOUT

FAIL

TIMEOUT

/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, persistent-usage-record, mp4, playback, retrieve in new window

TIMEOUT

FAIL

TIMEOUT


Misc:  Multiple key scenarios and one stress case.  Minor impact on proving out the spec.  Total failures: 3


/encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-different-mediakeys.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

org.w3.clearkey, setmediakeys multiple times with different mediakeys

PASS

FAIL

FAIL

/encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

update() with invalid response (longer than 64Kb characters) should fail.

PASS

FAIL

FAIL

/encrypted-media/drm-mp4-setmediakeys-multiple-times-with-different-mediakeys.html (1/1, 100.00%, 0.33% of total)

OK

OK

OK

drm, setmediakeys multiple times with different mediakeys

PASS

FAIL

FAIL

Received on Thursday, 13 October 2016 00:33:16 UTC