RE: EME Test Status Update - 10-24-16

I don’t see /encrypted-media/drm-mp4-playback-temporary-expired listed below.  Is it on your radar, or passing in your Nightly?

We’ve identified 14 test failures that must be resolved.  The issues you are working on important for that, so thanks Chris.

Here are the 14 and their current status:

Persistent-license failures - 5 Failures
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, destroy and acknowledge
Pass on Chrome OS

/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events
Pass on Chrome OS

/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4playback
Pass on Chrome OS

/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback and destroy
Pass on Chrome OS

/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback
Pass on Chrome OS

Persistent-usage-record failures - 3 Failures
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, check events
No current implementation

/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, retrieve in new window
No current implementation

/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4playback
No current implementation

Temporary session failures - 4 Failures
/encrypted-media/drm-mp4-playback-temporary-expired.html
drm, temporary, mp4, expired license
Status TBD

/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
WPT issue 4027, test as written cannot be passed with Widevine

/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential
WPT pull 3985 expected to fix

/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential
WPT pull 3985 expected to fix

Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential, readyState
EME issue #338 & #339 merged, should now pass on FF nightly

/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
EME issue #338 & #339 merged, should now pass on FF nightly

From: Chris Pearce [mailto:cpearce@mozilla.com]
Sent: Tuesday, October 25, 2016 3:06 PM
To: David Dorwin <ddorwin@google.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; public-html-media@w3.org
Subject: Re: EME Test Status Update - 10-24-16

OK, now that the above PR has landed, Firefox Nightly passes the readyState check. We're still failing the "Should not continue playing whilst waiting for a key expected" test, because it turns out that we don't correctly suspend playback when waiting for key. Good thing we've got a web platform test!
Chris Pearce.

On Wed, Oct 26, 2016 at 10:02 AM, David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> wrote:
https://github.com/w3c/web-platform-tests/pull/4076 fixes the readyState expectation. If you're happy with it, we can merge and you can address the playing count.

On Tue, Oct 25, 2016 at 2:01 PM, Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>> wrote:
I am working on updating PR https://github.com/w3c/web-platform-tests/pull/3985 now. This should make the .*sequential.* tests pass.

On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> wrote:
Some of these are likely due to a change in error type. See https://github.com/w3c/web-platform-tests/issues/4029#issuecomment-255476955 and https://github.com/w3c/web-platform-tests/issues/4030#issuecomment-255499074. (Chrome now passes the affected tests.)

For the readyState tests, we have spec issue https://github.com/w3c/encrypted-media/issues/336 and your open PR https://github.com/w3c/web-platform-tests/pull/3985. I'm also preparing a PR to update the tests for https://github.com/w3c/encrypted-media/issues/338. As discussed in the telecon this morning, we are hoping that Firefox will pass the (updated) readyState tests and demonstrate implementation since Chrome has unrelated bugs.

drm-temporary-license-type.html is incorrect and needs to be disabled or fixed. See https://github.com/w3c/web-platform-tests/issues/4027#issuecomment-255481770.


On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>> wrote:
Currently the set of tests I see failing in a local Firefox build other than persistent tests are these:

/encrypted-media/clearkey-invalid-license.html    FAIL        0/1
/encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html    NOTRUN        0/1
/encrypted-media/clearkey-mp4-syntax-mediakeysession.html    FAIL        6/7
/encrypted-media/clearkey-mp4-update-disallowed-input.html    FAIL        0/1
/encrypted-media/clearkey-update-non-ascii-input.html    FAIL        0/1
/encrypted-media/drm-invalid-license.html    FAIL        0/1
/encrypted-media/drm-mp4-playback-temporary-multisession.html    TIMEOUT        0/1
/encrypted-media/drm-mp4-syntax-mediakeysession.html    FAIL        6/7
/encrypted-media/drm-temporary-license-type.html    FAIL        0/1
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html    FAIL        0/1
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html    FAIL        0/1
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html    FAIL        0/1
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html    FAIL        0/1
Some of the tests have changed. We'll investigate.
Chris.


On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>> wrote:
David and Chris:

Do these test results include the implementation changes you previously announced in the following messages?
https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0044.html

https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0042.html


/paulc

From: Jerry Smith (WPT)
Sent: Monday, October 24, 2016 9:17 PM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>; Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; 'public-html-media@w3.org<mailto:public-html-media@w3.org>' <public-html-media@w3.org<mailto:public-html-media@w3.org>>
Subject: EME Test Status Update - 10-24-16

I’ve updated the test status reports to reflect recent fixes.  I believe some improvement was expected from last Thursday, but the results are largely the same.  I tested using:


-         Chrome 56.0.2899.0 canary (64 bit)

-         Firefox 52.0a1 (2016-10-24) (32 bit)

Reports:


•        http://w3c.github.io/test-results/encrypted-media/all.html


o   Test files: 97; Total subtests: 306

•        http://w3c.github.io/test-results/encrypted-media/complete-fails.html


o   Completely failed files: 21; Completely failed subtests: 21; Failure level: 21/306 (6.86%)

•        http://w3c.github.io/test-results/encrypted-media/less-than-2.html


o   Test files without 2 passes: 33; Subtests without 2 passes: 47; Failure level: 47/306 (15.36%)

Summary Groupings:

Errors (considered bugs, not blocking)                                  22
Clear Key persistent-license (not planned, may make non-normative)       5
Clear Key persistent-usage-record (not planned, at risk)                 3
Clear Key transitions between clear and encrypted (bugs, not blocking)   2

Non-Blocking issues                                                     32

Persistent-license (fixes in work)                                       6
Persistent-usage-record (no implementations in test, at risk)            3
Temporary session edge cases (bugs, not block, fixes in work)            3
Temporary session readyState (issues under discussion, at risk)          2

In work or at risk                                                      14

TOTAL                                                                   46

Details on failed tests are below.

Jerry


Error handling (considered bugs, not blocking) - 22 Failures
/encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33% of total)
Update with invalid Clear Key license
/encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7, 14.29%, 0.33% of total)
org.w3.clearkey test if MediaKeySession generateRequest() resolves for various sessions
/encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%, 0.33% of total)
update() with invalid response (longer than 64Kb characters) should fail.
/encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey test handling of non-ASCII responses for update()
/encrypted-media/drm-generate-request-disallowed-input.html (3/7, 42.86%, 0.98% of total)
drm, temporary, keyids, initData longer than 64Kb characters
drm, temporary, keyids, invalid initdata (too short key ID)
drm, temporary, keyids, invalid initdata (too long key ID)
/encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
Update with invalid Clear Key license
/encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%, 0.33% of total)
drm test if MediaKeySession generateRequest() resolves for various sessions
/encrypted-media/idlharness.ht<http://idlharness.ht>ml (10/58, 17.24%, 3.27% of total)
Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object Object])
HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
MediaKeySystemAccess interface: operation createMediaKeys()
MediaKeys interface: operation setServerCertificate(BufferSource)
MediaKeySession interface: operation generateRequest(DOMString,BufferSource)
MediaKeySession interface: operation load(DOMString)
MediaKeySession interface: operation update(BufferSource)
MediaKeySession interface: operation close()
MediaKeySession interface: operation remove()
MediaKeyMessageEvent interface object length
/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.ht<http://requestmediakeysystemaccess.ht>ml (1/58, 1.72%, 0.33% of total)
Trailing space in contentType
/encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%, 0.65% of total)
drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4; foo="bar"') should result in NotSupportedError
drm, requestMediaKeySystemAccess: Unrecognized foo with codecs ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in NotSupportedError

Clear key persistent-license tests (not planned to be implemented, may make informative note) - 5 Failures
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-license.ht<http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback

Clear key persistent-usage-record (not planned to be implemented) - 3 Failures
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window

Clear key transitions between clear and encrypted content (considered bugs, not blocking) - 2 Failures
/encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, temporary, mp4, playback, single key, clear then encrypted content
/encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, temporary, mp4, playback, single key, encrypted then clear content

Persistent-license failures - 6 Failures
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events
/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4playback
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback

Persistent-usage-record failures - 3 Failures
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, check events
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, retrieve in new window
/encrypted-media/drm-mp4-playback-persistent-usage-record.ht<http://ack-persistent-usage-record.ht>ml (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4playback

Temporary session failures - 3 Failures
/encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, expired license
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential

Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState

Received on Wednesday, 26 October 2016 00:21:18 UTC