Re: EME Test Status Update - 10-24-16

https://github.com/w3c/web-platform-tests/pull/4076 fixes the readyState
expectation. If you're happy with it, we can merge and you can address the
playing count.

On Tue, Oct 25, 2016 at 2:01 PM, Chris Pearce <cpearce@mozilla.com> wrote:

> I am working on updating PR https://github.com/w3c/web-
> platform-tests/pull/3985 now. This should make the .*sequential.* tests
> pass.
>
> On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com> wrote:
>
>> Some of these are likely due to a change in error type. See
>> https://github.com/w3c/web-platform-tests/issues/4029#is
>> suecomment-255476955 and https://github.com/w3c/web-platform
>> -tests/issues/4030#issuecomment-255499074. (Chrome now passes the
>> affected tests.)
>>
>> For the readyState tests, we have spec issue https://github.com/w3c/e
>> ncrypted-media/issues/336 and your open PR https://github.com/w3c/web-
>> platform-tests/pull/3985. I'm also preparing a PR to update the tests
>> for https://github.com/w3c/encrypted-media/issues/338. As discussed in
>> the telecon this morning, we are hoping that Firefox will pass the
>> (updated) readyState tests and demonstrate implementation since Chrome has
>> unrelated bugs.
>>
>> drm-temporary-license-type.html is incorrect and needs to be disabled or
>> fixed. See https://github.com/w3c/web-platform-tests/issues/4027#
>> issuecomment-255481770.
>>
>> On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com>
>> wrote:
>>
>>> Currently the set of tests I see failing in a local Firefox build other
>>> than persistent tests are these:
>>>
>>> /encrypted-media/clearkey-invalid-license.html    FAIL        0/1
>>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>>> NOTRUN        0/1
>>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html    FAIL
>>>     6/7
>>> /encrypted-media/clearkey-mp4-update-disallowed-input.html    FAIL
>>>     0/1
>>> /encrypted-media/clearkey-update-non-ascii-input.html    FAIL        0/1
>>> /encrypted-media/drm-invalid-license.html    FAIL        0/1
>>> /encrypted-media/drm-mp4-playback-temporary-multisession.html
>>> TIMEOUT        0/1
>>> /encrypted-media/drm-mp4-syntax-mediakeysession.html    FAIL        6/7
>>> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>>> FAIL        0/1
>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>>> FAIL        0/1
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>> FAIL        0/1
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>> FAIL        0/1
>>>
>>> Some of the tests have changed. We'll investigate.
>>>
>>> Chris.
>>>
>>>
>>> On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com>
>>> wrote:
>>>
>>>> David and Chris:
>>>>
>>>>
>>>>
>>>> Do these test results include the implementation changes you previously
>>>> announced in the following messages?
>>>>
>>>> https://lists.w3.org/Archives/Public/public-html-media/2016O
>>>> ct/0044.html
>>>>
>>>> https://lists.w3.org/Archives/Public/public-html-media/2016O
>>>> ct/0042.html
>>>>
>>>>
>>>>
>>>> /paulc
>>>>
>>>>
>>>>
>>>> *From:* Jerry Smith (WPT)
>>>> *Sent:* Monday, October 24, 2016 9:17 PM
>>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>>>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>>>> wolenetz@google.com>; 'public-html-media@w3.org' <
>>>> public-html-media@w3.org>
>>>> *Subject:* EME Test Status Update - 10-24-16
>>>>
>>>>
>>>>
>>>> I’ve updated the test status reports to reflect recent fixes.  I
>>>> believe some improvement was expected from last Thursday, but the results
>>>> are largely the same.  I tested using:
>>>>
>>>>
>>>>
>>>> -         Chrome 56.0.2899.0 canary (64 bit)
>>>>
>>>> -         Firefox 52.0a1 (2016-10-24) (32 bit)
>>>>
>>>>
>>>>
>>>> Reports:
>>>>
>>>>
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>>>
>>>> o   *Test files*: 97; *Total subtests*: 306
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/complete-f
>>>> ails.html
>>>>
>>>> o   *Completely failed files*: 21; *Completely failed subtests*: 21; *Failure
>>>> level*: 21/306 (6.86%)
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>>>> 2.html
>>>>
>>>> o   *Test files without 2 passes*: 33; *Subtests without 2 passes: *47;
>>>> *Failure level*: 47/306 (15.36%)
>>>>
>>>>
>>>>
>>>> Summary Groupings:
>>>>
>>>>
>>>>
>>>> Errors (considered bugs, not blocking)
>>>> 22
>>>>
>>>> Clear Key persistent-license (not planned, may make
>>>> non-normative)       5
>>>>
>>>> Clear Key persistent-usage-record (not planned, at
>>>> risk)                 3
>>>>
>>>> Clear Key transitions between clear and encrypted (bugs, not
>>>> blocking)   2
>>>>
>>>>
>>>>
>>>> Non-Blocking issues
>>>>                              32
>>>>
>>>>
>>>>
>>>> Persistent-license (fixes in work)
>>>>             6
>>>>
>>>> Persistent-usage-record (no implementations in test, at
>>>> risk)            3
>>>>
>>>> Temporary session edge cases (bugs, not block, fixes in
>>>> work)            3
>>>>
>>>> Temporary session readyState (issues under discussion, at
>>>> risk)          2
>>>>
>>>>
>>>>
>>>> In work or at risk
>>>> 14
>>>>
>>>>
>>>>
>>>> TOTAL
>>>> 46
>>>>
>>>>
>>>>
>>>> Details on failed tests are below.
>>>>
>>>>
>>>>
>>>> Jerry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Error handling (considered bugs, not blocking) - 22 Failures
>>>>
>>>> /encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33% of
>>>> total)
>>>>
>>>> Update with invalid Clear Key license
>>>>
>>>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7,
>>>> 14.29%, 0.33% of total)
>>>>
>>>> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
>>>> various sessions
>>>>
>>>> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> update() with invalid response (longer than 64Kb characters) should
>>>> fail.
>>>>
>>>> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
>>>> 0.33% of total)
>>>>
>>>> org.w3.clearkey test handling of non-ASCII responses for update()
>>>>
>>>> /encrypted-media/drm-generate-request-disallowed-input.html (3/7,
>>>> 42.86%, 0.98% of total)
>>>>
>>>> drm, temporary, keyids, initData longer than 64Kb characters
>>>>
>>>> drm, temporary, keyids, invalid initdata (too short key ID)
>>>>
>>>> drm, temporary, keyids, invalid initdata (too long key ID)
>>>>
>>>> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of
>>>> total)
>>>>
>>>> Update with invalid Clear Key license
>>>>
>>>> /encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%,
>>>> 0.33% of total)
>>>>
>>>> drm test if MediaKeySession generateRequest() resolves for various
>>>> sessions
>>>>
>>>> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>>>>
>>>> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
>>>> Object])
>>>>
>>>> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>>>>
>>>> MediaKeySystemAccess interface: operation createMediaKeys()
>>>>
>>>> MediaKeys interface: operation setServerCertificate(BufferSource)
>>>>
>>>> MediaKeySession interface: operation generateRequest(DOMString,Buff
>>>> erSource)
>>>>
>>>> MediaKeySession interface: operation load(DOMString)
>>>>
>>>> MediaKeySession interface: operation update(BufferSource)
>>>>
>>>> MediaKeySession interface: operation close()
>>>>
>>>> MediaKeySession interface: operation remove()
>>>>
>>>> MediaKeyMessageEvent interface object length
>>>>
>>>> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
>>>> 1.72%, 0.33% of total)
>>>>
>>>> Trailing space in contentType
>>>>
>>>> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58,
>>>> 3.45%, 0.65% of total)
>>>>
>>>> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4
>>>> ('video/mp4; foo="bar"') should result in NotSupportedError
>>>>
>>>> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
>>>> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
>>>> NotSupportedError
>>>>
>>>>
>>>>
>>>> Clear key persistent-license tests (not planned to be implemented, may
>>>> make informative note) - 5 Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>>>> acknowledge
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4playback
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>>>> and destroy
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and
>>>> playback
>>>>
>>>>
>>>>
>>>> Clear key persistent-usage-record (not planned to be implemented) - 3
>>>> Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4playback
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in
>>>> new window
>>>>
>>>>
>>>>
>>>> Clear key transitions between clear and encrypted content (considered
>>>> bugs, not blocking) - 2 Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, temporary, mp4, playback, single key, clear then
>>>> encrypted content
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
>>>> clear content
>>>>
>>>>
>>>>
>>>> Persistent-license failures - 6 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, destroy and acknowledge
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, check events
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4playback
>>>>
>>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>>> of total)
>>>>
>>>> drm, cannot load persistent license into temporary session
>>>>
>>>> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>>>>
>>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, retrieve, playback
>>>>
>>>>
>>>>
>>>> Persistent-usage-record failures - 3 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4, playback, check events
>>>>
>>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4playback
>>>>
>>>>
>>>>
>>>> Temporary session failures - 3 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, temporary, mp4, expired license
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>>> sequential
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>>
>>>>
>>>>
>>>> Temporary session readyState failures (listed at risk, under
>>>> discussion) - 2 Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>>> sequential, readyState
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>>> readyState
>>>>
>>>
>>>
>>
>