Re: EME Test Status Update - 10-24-16

OK, now that the above PR has landed, Firefox Nightly passes the readyState
check. We're still failing the "Should not continue playing whilst waiting
for a key expected" test, because it turns out that we don't correctly
suspend playback when waiting for key. Good thing we've got a web platform
test!

Chris Pearce.

On Wed, Oct 26, 2016 at 10:02 AM, David Dorwin <ddorwin@google.com> wrote:

> https://github.com/w3c/web-platform-tests/pull/4076 fixes the readyState
> expectation. If you're happy with it, we can merge and you can address the
> playing count.
>
> On Tue, Oct 25, 2016 at 2:01 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
>> I am working on updating PR https://github.com/w3c/web-
>> platform-tests/pull/3985 now. This should make the .*sequential.* tests
>> pass.
>>
>> On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com> wrote:
>>
>>> Some of these are likely due to a change in error type. See
>>> https://github.com/w3c/web-platform-tests/issues/4029#is
>>> suecomment-255476955 and https://github.com/w3c/web-platform
>>> -tests/issues/4030#issuecomment-255499074. (Chrome now passes the
>>> affected tests.)
>>>
>>> For the readyState tests, we have spec issue https://github.com/w3c/e
>>> ncrypted-media/issues/336 and your open PR https://github.com/w3c/web-
>>> platform-tests/pull/3985. I'm also preparing a PR to update the tests
>>> for https://github.com/w3c/encrypted-media/issues/338. As discussed in
>>> the telecon this morning, we are hoping that Firefox will pass the
>>> (updated) readyState tests and demonstrate implementation since Chrome has
>>> unrelated bugs.
>>>
>>> drm-temporary-license-type.html is incorrect and needs to be disabled
>>> or fixed. See https://github.com/w3c/web-platform-tests/issues/4027#is
>>> suecomment-255481770.
>>>
>>> On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com>
>>> wrote:
>>>
>>>> Currently the set of tests I see failing in a local Firefox build other
>>>> than persistent tests are these:
>>>>
>>>> /encrypted-media/clearkey-invalid-license.html    FAIL        0/1
>>>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>>>> NOTRUN        0/1
>>>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html    FAIL
>>>>     6/7
>>>> /encrypted-media/clearkey-mp4-update-disallowed-input.html    FAIL
>>>>     0/1
>>>> /encrypted-media/clearkey-update-non-ascii-input.html    FAIL
>>>> 0/1
>>>> /encrypted-media/drm-invalid-license.html    FAIL        0/1
>>>> /encrypted-media/drm-mp4-playback-temporary-multisession.html
>>>> TIMEOUT        0/1
>>>> /encrypted-media/drm-mp4-syntax-mediakeysession.html    FAIL        6/7
>>>> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> FAIL        0/1
>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>>>> FAIL        0/1
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> FAIL        0/1
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>>> FAIL        0/1
>>>>
>>>> Some of the tests have changed. We'll investigate.
>>>>
>>>> Chris.
>>>>
>>>>
>>>> On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com
>>>> > wrote:
>>>>
>>>>> David and Chris:
>>>>>
>>>>>
>>>>>
>>>>> Do these test results include the implementation changes you
>>>>> previously announced in the following messages?
>>>>>
>>>>> https://lists.w3.org/Archives/Public/public-html-media/2016O
>>>>> ct/0044.html
>>>>>
>>>>> https://lists.w3.org/Archives/Public/public-html-media/2016O
>>>>> ct/0042.html
>>>>>
>>>>>
>>>>>
>>>>> /paulc
>>>>>
>>>>>
>>>>>
>>>>> *From:* Jerry Smith (WPT)
>>>>> *Sent:* Monday, October 24, 2016 9:17 PM
>>>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>>>>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz
>>>>> <wolenetz@google.com>; 'public-html-media@w3.org' <
>>>>> public-html-media@w3.org>
>>>>> *Subject:* EME Test Status Update - 10-24-16
>>>>>
>>>>>
>>>>>
>>>>> I’ve updated the test status reports to reflect recent fixes.  I
>>>>> believe some improvement was expected from last Thursday, but the results
>>>>> are largely the same.  I tested using:
>>>>>
>>>>>
>>>>>
>>>>> -         Chrome 56.0.2899.0 canary (64 bit)
>>>>>
>>>>> -         Firefox 52.0a1 (2016-10-24) (32 bit)
>>>>>
>>>>>
>>>>>
>>>>> Reports:
>>>>>
>>>>>
>>>>>
>>>>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>>>>
>>>>> o   *Test files*: 97; *Total subtests*: 306
>>>>>
>>>>> ·        http://w3c.github.io/test-results/encrypted-media/complete-f
>>>>> ails.html
>>>>>
>>>>> o   *Completely failed files*: 21; *Completely failed subtests*: 21; *Failure
>>>>> level*: 21/306 (6.86%)
>>>>>
>>>>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>>>>> 2.html
>>>>>
>>>>> o   *Test files without 2 passes*: 33; *Subtests without 2 passes: *47;
>>>>> *Failure level*: 47/306 (15.36%)
>>>>>
>>>>>
>>>>>
>>>>> Summary Groupings:
>>>>>
>>>>>
>>>>>
>>>>> Errors (considered bugs, not blocking)
>>>>> 22
>>>>>
>>>>> Clear Key persistent-license (not planned, may make
>>>>> non-normative)       5
>>>>>
>>>>> Clear Key persistent-usage-record (not planned, at
>>>>> risk)                 3
>>>>>
>>>>> Clear Key transitions between clear and encrypted (bugs, not
>>>>> blocking)   2
>>>>>
>>>>>
>>>>>
>>>>> Non-Blocking issues
>>>>>                              32
>>>>>
>>>>>
>>>>>
>>>>> Persistent-license (fixes in work)
>>>>>             6
>>>>>
>>>>> Persistent-usage-record (no implementations in test, at
>>>>> risk)            3
>>>>>
>>>>> Temporary session edge cases (bugs, not block, fixes in
>>>>> work)            3
>>>>>
>>>>> Temporary session readyState (issues under discussion, at
>>>>> risk)          2
>>>>>
>>>>>
>>>>>
>>>>> In work or at risk
>>>>> 14
>>>>>
>>>>>
>>>>>
>>>>> TOTAL
>>>>> 46
>>>>>
>>>>>
>>>>>
>>>>> Details on failed tests are below.
>>>>>
>>>>>
>>>>>
>>>>> Jerry
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Error handling (considered bugs, not blocking) - 22 Failures
>>>>>
>>>>> /encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33%
>>>>> of total)
>>>>>
>>>>> Update with invalid Clear Key license
>>>>>
>>>>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7,
>>>>> 14.29%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
>>>>> various sessions
>>>>>
>>>>> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1,
>>>>> 100.00%, 0.33% of total)
>>>>>
>>>>> update() with invalid response (longer than 64Kb characters) should
>>>>> fail.
>>>>>
>>>>> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
>>>>> 0.33% of total)
>>>>>
>>>>> org.w3.clearkey test handling of non-ASCII responses for update()
>>>>>
>>>>> /encrypted-media/drm-generate-request-disallowed-input.html (3/7,
>>>>> 42.86%, 0.98% of total)
>>>>>
>>>>> drm, temporary, keyids, initData longer than 64Kb characters
>>>>>
>>>>> drm, temporary, keyids, invalid initdata (too short key ID)
>>>>>
>>>>> drm, temporary, keyids, invalid initdata (too long key ID)
>>>>>
>>>>> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of
>>>>> total)
>>>>>
>>>>> Update with invalid Clear Key license
>>>>>
>>>>> /encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%,
>>>>> 0.33% of total)
>>>>>
>>>>> drm test if MediaKeySession generateRequest() resolves for various
>>>>> sessions
>>>>>
>>>>> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>>>>>
>>>>> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
>>>>> Object])
>>>>>
>>>>> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>>>>>
>>>>> MediaKeySystemAccess interface: operation createMediaKeys()
>>>>>
>>>>> MediaKeys interface: operation setServerCertificate(BufferSource)
>>>>>
>>>>> MediaKeySession interface: operation generateRequest(DOMString,Buff
>>>>> erSource)
>>>>>
>>>>> MediaKeySession interface: operation load(DOMString)
>>>>>
>>>>> MediaKeySession interface: operation update(BufferSource)
>>>>>
>>>>> MediaKeySession interface: operation close()
>>>>>
>>>>> MediaKeySession interface: operation remove()
>>>>>
>>>>> MediaKeyMessageEvent interface object length
>>>>>
>>>>> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
>>>>> 1.72%, 0.33% of total)
>>>>>
>>>>> Trailing space in contentType
>>>>>
>>>>> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58,
>>>>> 3.45%, 0.65% of total)
>>>>>
>>>>> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4
>>>>> ('video/mp4; foo="bar"') should result in NotSupportedError
>>>>>
>>>>> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
>>>>> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
>>>>> NotSupportedError
>>>>>
>>>>>
>>>>>
>>>>> Clear key persistent-license tests (not planned to be implemented, may
>>>>> make informative note) - 5 Failures
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>>>>> acknowledge
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>>>>> 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-license, mp4playback
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>>>>> and destroy
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and
>>>>> playback
>>>>>
>>>>>
>>>>>
>>>>> Clear key persistent-usage-record (not planned to be implemented) - 3
>>>>> Failures
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-usage-record, mp4playback
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in
>>>>> new window
>>>>>
>>>>>
>>>>>
>>>>> Clear key transitions between clear and encrypted content (considered
>>>>> bugs, not blocking) - 2 Failures
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, temporary, mp4, playback, single key, clear then
>>>>> encrypted content
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
>>>>> clear content
>>>>>
>>>>>
>>>>>
>>>>> Persistent-license failures - 6 Failures
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-license, mp4, playback, destroy and acknowledge
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-license, mp4, playback, check events
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1,
>>>>> 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-license, mp4playback
>>>>>
>>>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>>>> of total)
>>>>>
>>>>> drm, cannot load persistent license into temporary session
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-license, mp4, playback, retrieve, playback
>>>>>
>>>>>
>>>>>
>>>>> Persistent-usage-record failures - 3 Failures
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-usage-record, mp4, playback, check events
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>>>>> 100.00%, 0.33% of total)
>>>>>
>>>>> drm, persistent-usage-record, mp4playback
>>>>>
>>>>>
>>>>>
>>>>> Temporary session failures - 3 Failures
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1,
>>>>> 100.00%, 0.33% of total)
>>>>>
>>>>> drm, temporary, mp4, expired license
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>>>> sequential
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>>>
>>>>>
>>>>>
>>>>> Temporary session readyState failures (listed at risk, under
>>>>> discussion) - 2 Failures
>>>>>
>>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>>>> sequential, readyState
>>>>>
>>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>>>> (1/1, 100.00%, 0.33% of total)
>>>>>
>>>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>>>> readyState
>>>>>
>>>>
>>>>
>>>
>>
>