Re: EME test status

Mozilla is working on making the EME Web Platform Tests in the "Wrong Error
Handling", "Temporary Session", and "Misc" categories pass in Firefox 52,
which is in our Nightly release channel. We have submitted several pull
requests for bugs in the tests, fixed a number of issues in Firefox, and
the number of tests that pass in these categories in Firefox Nightly is
increasing daily.

At this time we are not planning on implementing "persistent-usage-record"
or "persistent-license" sessions. We won't be trying to make the
"Persistent-License Sessions" and "Persistent-Usage-Record" tests pass in
the EME v1 time frame.

Cheers,
Chris Pearce.



On Thu, Oct 13, 2016 at 1:32 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> There were some minor bug fixes in the text suite today.  I’ve run the
> tests and posted reports:
>
>
>
> -                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/all.html
>
> o   *Test files*: 97; *Total subtests*: 306
>
> -                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/less-than-2.html
>
> o   *Test files without 2 passes*: 37; *Subtests without 2 passes: *68; *Failure
> level*: 68/306 (22.22%)
>
> -                 https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/complete-fails.html
>
> o   *Completely failed files*: undefined; *Completely failed subtests*:
> 44; *Failure level*: 44/306 (14.38%)
>
>
>
> Our test suite work is scheduled to be done on October 25th, so we have
> just under 2 weeks remaining.  Given the limited time remaining, I
> recommend we focus on closures and not on adding any new tests.  I’ve
> grouped the less than 2 results below to help draw some conclusions using
> these groupings:
>
>
>
> -          Wrong Error Handling:                 42 failures are this
> type (out of 68 total failures).  It would be great to do an implementation
> push to get these passing, since conceptually it would leave only 26
> remaining failures/timeouts.  The majority of these are returning the wrong
> error.
>
> -          Temporary Session:                      6 of these are failing
> and I believe some are already being investigated.  Can they be closed by
> next week?
>
> -          Persistent-License Sessions:        11 (all tests) are
> failing, which suggests that Persistent-License should be considered “at
> risk”.  A few of these pass with Polyfills.
>
> -          Persistent-Usage-Record:            6 (all tests) are
> failing.  This feature is already listed at risk in the current CR.
>
> -          Misc:                                              3 tests
> don’t really map to any of the above.
>
>
>
> It would be good to confirm:
>
>
>
> -          Whether Chrome and Firefox can make fixes to close the Error
> handling failures.
>
> -          What the status is on the 6 Temporary session bug
> investigations.
>
> -          Whether any work is planned on Persistent-License failures.
>
> -          Whether any work is planned on Persistent-Usage-Record
> failures.
>
>
>
> Test groupings are below.
>
>
>
> Jerry
>
>
>
> *Test*
>
> *CH56*
>
> *ED14*
>
> *FF52*
>
> *Error Handling:  These failures are a result of either returning the
> wrong error type, or not properly failing a promise after an error.  Total
> failures:  42*
>
> /encrypted-media/clearkey-generate-request-disallowed-input.html (7/7,
> 100.00%, 2.29% of total)
>
> OK
>
> TIMEOUT
>
> OK
>
> org.w3.clearkey, temporary, webm, initData longer than 64Kb characters
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, cenc, initData longer than 64Kb characters
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, keyids, initData longer than 64Kb characters
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, cenc, invalid initdata (invalid pssh)
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, cenc, invalid initdata (not pssh)
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, keyids, invalid initdata (too short key ID)
>
> FAIL
>
> -
>
> FAIL
>
> org.w3.clearkey, temporary, keyids, invalid initdata (too long key ID)
>
> FAIL
>
> -
>
> FAIL
>
> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (4/7, 57.14%,
> 1.31% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey test MediaKeySession generateRequest() exceptions.
>
> PASS
>
> FAIL
>
> FAIL
>
> org.w3.clearkey test MediaKeySession load() exceptions.
>
> FAIL
>
> FAIL
>
> FAIL
>
> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
> various sessions
>
> PASS
>
> FAIL
>
> FAIL
>
> org.w3.clearkey test MediaKeySession update() exceptions.
>
> PASS
>
> FAIL
>
> FAIL
>
> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (3/58,
> 5.17%, 0.98% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, requestMediaKeySystemAccess: Empty Key System should
> result in TypeError
>
> FAIL
>
> FAIL
>
> FAIL
>
> org.w3.clearkey, requestMediaKeySystemAccess: Empty
> supportedConfigurations should result in TypeError
>
> FAIL
>
> FAIL
>
> FAIL
>
> Trailing space in contentType
>
> FAIL
>
> FAIL
>
> PASS
>
> /encrypted-media/clearkey-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33%
> of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey test MediaKeys setServerCertificate() exceptions.
>
> FAIL
>
> FAIL
>
> PASS
>
> /encrypted-media/clearkey-mp4-syntax-mediakeysystemaccess.html (1/2,
> 50.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey test requestMediaKeySystemAccess() exceptions.
>
> FAIL
>
> FAIL
>
> FAIL
>
> /encrypted-media/drm-generate-request-disallowed-input.html (7/7,
> 100.00%, 2.29% of total)
>
> OK
>
> OK
>
> OK
>
> drm, temporary, webm, initData longer than 64Kb characters
>
> FAIL
>
> -
>
> FAIL
>
> drm, temporary, cenc, initData longer than 64Kb characters
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm, temporary, cenc, invalid initdata (invalid pssh)
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm, temporary, cenc, invalid initdata (not pssh)
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm, temporary, keyids, initData longer than 64Kb characters
>
> -
>
> FAIL
>
> FAIL
>
> drm, temporary, keyids, invalid initdata (too short key ID)
>
> -
>
> FAIL
>
> FAIL
>
> drm, temporary, keyids, invalid initdata (too long key ID)
>
> -
>
> FAIL
>
> FAIL
>
> /encrypted-media/drm-mp4-syntax-mediakeysession.html (3/7, 42.86%, 0.98%
> of total)
>
> OK
>
> OK
>
> OK
>
> drm test MediaKeySession generateRequest() exceptions.
>
> PASS
>
> FAIL
>
> FAIL
>
> drm test MediaKeySession load() exceptions.
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm test MediaKeySession update() exceptions.
>
> PASS
>
> FAIL
>
> FAIL
>
> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (4/58, 6.90%,
> 1.31% of total)
>
> OK
>
> OK
>
> OK
>
> drm, requestMediaKeySystemAccess: Empty Key System should result in
> TypeError
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm, requestMediaKeySystemAccess: Empty supportedConfigurations should
> result in TypeError
>
> FAIL
>
> FAIL
>
> FAIL
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4;
> foo="bar"') should result in NotSupportedError
>
> FAIL
>
> FAIL
>
> PASS
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
> NotSupportedError
>
> FAIL
>
> FAIL
>
> PASS
>
> /encrypted-media/drm-mp4-syntax-mediakeys.html (1/3, 33.33%, 0.33% of
> total)
>
> OK
>
> OK
>
> OK
>
> drm test MediaKeys setServerCertificate() exceptions.
>
> FAIL
>
> FAIL
>
> PASS
>
> /encrypted-media/drm-mp4-syntax-mediakeysystemaccess.html (1/2, 50.00%,
> 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm test requestMediaKeySystemAccess() exceptions.
>
> FAIL
>
> FAIL
>
> FAIL
>
> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>
> OK
>
> OK
>
> OK
>
> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
> Object])
>
> FAIL
>
> FAIL
>
> PASS
>
> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySystemAccess interface: operation createMediaKeys()
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeys interface: operation setServerCertificate(BufferSource)
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySession interface: operation generateRequest(DOMString,
> BufferSource)
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySession interface: operation load(DOMString)
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySession interface: operation update(BufferSource)
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySession interface: operation close()
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeySession interface: operation remove()
>
> FAIL
>
> FAIL
>
> PASS
>
> MediaKeyMessageEvent interface object length
>
> FAIL
>
> FAIL
>
> FAIL
>
> *Temporary Sessions:  These failures are in tests that evaluate specific
> protected content scenarios.  Many temporary sessions are supported today,
> but not these variants.  Total failures: 6  *
>
> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, temporary, mp4, playback, single key, clear then
> encrypted content
>
> TIMEOUT
>
> TIMEOUT
>
> PASS
>
> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
> clear content
>
> TIMEOUT
>
> TIMEOUT
>
> PASS
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%,
> 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, temporary, mp4, expired license
>
> FAIL
>
> FAIL
>
> FAIL
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential, readyState
>
> FAIL
>
> TIMEOUT
>
> FAIL
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> TIMEOUT
>
> OK
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> FAIL
>
> -
>
> FAIL
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> TIMEOUT
>
> OK
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
> FAIL
>
> -
>
> FAIL
>
> *Persistent-License Sessions:  These failures are in tests that evaluate
> persistent license scenarios.  These are largely unsupported today.  Total
> failures: 11*
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
>
> POLYFILL
>
> TIMEOUT
>
> POLYFILL
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-license, mp4, playback, check events
>
> POLYFILL
>
> TIMEOUT
>
> POLYFILL
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-license, mp4playback
>
> POLYFILL
>
> TIMEOUT
>
> POLYFILL
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> OK
>
> OK
>
> OK
>
> drm, cannot load persistent license into temporary session
>
> FAIL
>
> FAIL
>
> FAIL
>
> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-license, mp4, playback, destroy and acknowledge
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-license, mp4, playback, check events
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-license, mp4playback
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-license, mp4, playback, retrieve, playback
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> *Persistent-Usage-Record Sessions:  These failures are in tests that
> evaluate persistent usage record scenarios.  These are largely unsupported
> today.  Total failures: 6*
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> POLYFILL
>
> TIMEOUT
>
> POLYFILL
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-usage-record, mp4playback
>
> POLYFILL
>
> TIMEOUT
>
> POLYFILL
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
> window
>
> TIMEOUT
>
> TIMEOUT
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-usage-record, mp4playback
>
> TIMEOUT
>
> FAIL
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-usage-record, mp4, playback, check events
>
> TIMEOUT
>
> FAIL
>
> TIMEOUT
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> TIMEOUT
>
> FAIL
>
> TIMEOUT
>
> *Misc:  Multiple key scenarios and one stress case.  Minor impact on
> proving out the spec.  Total failures: 3*
>
> /encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-different-mediakeys.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> org.w3.clearkey, setmediakeys multiple times with different mediakeys
>
> PASS
>
> FAIL
>
> FAIL
>
> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%,
> 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> update() with invalid response (longer than 64Kb characters) should fail.
>
> PASS
>
> FAIL
>
> FAIL
>
> /encrypted-media/drm-mp4-setmediakeys-multiple-times-with-different-mediakeys.html
> (1/1, 100.00%, 0.33% of total)
>
> OK
>
> OK
>
> OK
>
> drm, setmediakeys multiple times with different mediakeys
>
> PASS
>
> FAIL
>
> FAIL
>
>
>