Re: EME Test Status Update - 10-24-16

I am working on updating PR https://github.com/w3c/web-
platform-tests/pull/3985 now. This should make the .*sequential.* tests
pass.

On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com> wrote:

> Some of these are likely due to a change in error type. See
> https://github.com/w3c/web-platform-tests/issues/4029#
> issuecomment-255476955 and https://github.com/w3c/web-
> platform-tests/issues/4030#issuecomment-255499074. (Chrome now passes the
> affected tests.)
>
> For the readyState tests, we have spec issue https://github.com/w3c/e
> ncrypted-media/issues/336 and your open PR https://github.com/w3c/web-
> platform-tests/pull/3985. I'm also preparing a PR to update the tests for
> https://github.com/w3c/encrypted-media/issues/338. As discussed in the
> telecon this morning, we are hoping that Firefox will pass the (updated)
> readyState tests and demonstrate implementation since Chrome has unrelated
> bugs.
>
> drm-temporary-license-type.html is incorrect and needs to be disabled or
> fixed. See https://github.com/w3c/web-platform-tests/issues/
> 4027#issuecomment-255481770.
>
> On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
>> Currently the set of tests I see failing in a local Firefox build other
>> than persistent tests are these:
>>
>> /encrypted-media/clearkey-invalid-license.html    FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>> NOTRUN        0/1
>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html    FAIL
>> 6/7
>> /encrypted-media/clearkey-mp4-update-disallowed-input.html    FAIL
>>     0/1
>> /encrypted-media/clearkey-update-non-ascii-input.html    FAIL        0/1
>> /encrypted-media/drm-invalid-license.html    FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multisession.html
>> TIMEOUT        0/1
>> /encrypted-media/drm-mp4-syntax-mediakeysession.html    FAIL        6/7
>> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>> FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>> FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> FAIL        0/1
>>
>> Some of the tests have changed. We'll investigate.
>>
>> Chris.
>>
>>
>> On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com>
>> wrote:
>>
>>> David and Chris:
>>>
>>>
>>>
>>> Do these test results include the implementation changes you previously
>>> announced in the following messages?
>>>
>>> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0044.html
>>>
>>> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0042.html
>>>
>>>
>>>
>>> /paulc
>>>
>>>
>>>
>>> *From:* Jerry Smith (WPT)
>>> *Sent:* Monday, October 24, 2016 9:17 PM
>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>>> wolenetz@google.com>; 'public-html-media@w3.org' <
>>> public-html-media@w3.org>
>>> *Subject:* EME Test Status Update - 10-24-16
>>>
>>>
>>>
>>> I’ve updated the test status reports to reflect recent fixes.  I believe
>>> some improvement was expected from last Thursday, but the results are
>>> largely the same.  I tested using:
>>>
>>>
>>>
>>> -         Chrome 56.0.2899.0 canary (64 bit)
>>>
>>> -         Firefox 52.0a1 (2016-10-24) (32 bit)
>>>
>>>
>>>
>>> Reports:
>>>
>>>
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>>
>>> o   *Test files*: 97; *Total subtests*: 306
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/complete-f
>>> ails.html
>>>
>>> o   *Completely failed files*: 21; *Completely failed subtests*: 21; *Failure
>>> level*: 21/306 (6.86%)
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>>> 2.html
>>>
>>> o   *Test files without 2 passes*: 33; *Subtests without 2 passes: *47; *Failure
>>> level*: 47/306 (15.36%)
>>>
>>>
>>>
>>> Summary Groupings:
>>>
>>>
>>>
>>> Errors (considered bugs, not blocking)
>>> 22
>>>
>>> Clear Key persistent-license (not planned, may make non-normative)
>>> 5
>>>
>>> Clear Key persistent-usage-record (not planned, at risk)
>>> 3
>>>
>>> Clear Key transitions between clear and encrypted (bugs, not blocking)
>>> 2
>>>
>>>
>>>
>>> Non-Blocking issues                                                     3
>>> 2
>>>
>>>
>>>
>>> Persistent-license (fixes in work)
>>>             6
>>>
>>> Persistent-usage-record (no implementations in test, at risk)
>>> 3
>>>
>>> Temporary session edge cases (bugs, not block, fixes in work)
>>> 3
>>>
>>> Temporary session readyState (issues under discussion, at risk)
>>> 2
>>>
>>>
>>>
>>> In work or at risk
>>> 14
>>>
>>>
>>>
>>> TOTAL
>>> 46
>>>
>>>
>>>
>>> Details on failed tests are below.
>>>
>>>
>>>
>>> Jerry
>>>
>>>
>>>
>>>
>>>
>>> Error handling (considered bugs, not blocking) - 22 Failures
>>>
>>> /encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33% of
>>> total)
>>>
>>> Update with invalid Clear Key license
>>>
>>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7, 14.29%,
>>> 0.33% of total)
>>>
>>> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
>>> various sessions
>>>
>>> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> update() with invalid response (longer than 64Kb characters) should fail.
>>>
>>> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
>>> 0.33% of total)
>>>
>>> org.w3.clearkey test handling of non-ASCII responses for update()
>>>
>>> /encrypted-media/drm-generate-request-disallowed-input.html (3/7,
>>> 42.86%, 0.98% of total)
>>>
>>> drm, temporary, keyids, initData longer than 64Kb characters
>>>
>>> drm, temporary, keyids, invalid initdata (too short key ID)
>>>
>>> drm, temporary, keyids, invalid initdata (too long key ID)
>>>
>>> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
>>>
>>> Update with invalid Clear Key license
>>>
>>> /encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%,
>>> 0.33% of total)
>>>
>>> drm test if MediaKeySession generateRequest() resolves for various
>>> sessions
>>>
>>> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>>>
>>> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
>>> Object])
>>>
>>> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>>>
>>> MediaKeySystemAccess interface: operation createMediaKeys()
>>>
>>> MediaKeys interface: operation setServerCertificate(BufferSource)
>>>
>>> MediaKeySession interface: operation generateRequest(DOMString,Buff
>>> erSource)
>>>
>>> MediaKeySession interface: operation load(DOMString)
>>>
>>> MediaKeySession interface: operation update(BufferSource)
>>>
>>> MediaKeySession interface: operation close()
>>>
>>> MediaKeySession interface: operation remove()
>>>
>>> MediaKeyMessageEvent interface object length
>>>
>>> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
>>> 1.72%, 0.33% of total)
>>>
>>> Trailing space in contentType
>>>
>>> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%,
>>> 0.65% of total)
>>>
>>> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4;
>>> foo="bar"') should result in NotSupportedError
>>>
>>> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
>>> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
>>> NotSupportedError
>>>
>>>
>>>
>>> Clear key persistent-license tests (not planned to be implemented, may
>>> make informative note) - 5 Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>>> acknowledge
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4playback
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>>> and destroy
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>>>
>>>
>>>
>>> Clear key persistent-usage-record (not planned to be implemented) - 3
>>> Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4playback
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
>>> window
>>>
>>>
>>>
>>> Clear key transitions between clear and encrypted content (considered
>>> bugs, not blocking) - 2 Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, temporary, mp4, playback, single key, clear then
>>> encrypted content
>>>
>>> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
>>> clear content
>>>
>>>
>>>
>>> Persistent-license failures - 6 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, destroy and acknowledge
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, check events
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4playback
>>>
>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>> of total)
>>>
>>> drm, cannot load persistent license into temporary session
>>>
>>> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>>>
>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, retrieve, playback
>>>
>>>
>>>
>>> Persistent-usage-record failures - 3 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4, playback, check events
>>>
>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4playback
>>>
>>>
>>>
>>> Temporary session failures - 3 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%,
>>> 0.33% of total)
>>>
>>> drm, temporary, mp4, expired license
>>>
>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>> sequential
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>
>>>
>>>
>>> Temporary session readyState failures (listed at risk, under discussion)
>>> - 2 Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>>> sequential, readyState
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>> readyState
>>>
>>
>>
>