Re: EME Test Status Update - 10-24-16

/encrypted-media/drm-mp4-playback-temporary-expired.html
<https://w3c-test.org/encrypted-media/drm-mp4-playback-temporary-expired.html>
is an intermittent failure caused (at least for us here in NZ) by the
download of test resources taking 7 seconds, and the rest of the test then
not being able to complete within the default 10 second test time limit.
Extending the test time limit makes the test pass more reliably.

Fixed in PR:
https://github.com/w3c/web-platform-tests/pull/4080


On Wed, Oct 26, 2016 at 1:48 PM, Chris Pearce <cpearce@mozilla.com> wrote:

>
>
> On Wed, Oct 26, 2016 at 1:20 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
>> I don’t see /encrypted-media/drm-mp4-playback-temporary-expired listed
>> below.  Is it on your radar, or passing in your Nightly?
>>
>
>
> encrypted-media/drm-mp4-playback-temporary-expired
> <http://w3c-test.org/encrypted-media/drm-mp4-playback-temporary-expired.html>
> is intermittently passing about half the time in Firefox Nightly on my
> local machine. We're looking into this one still.
>
>
>
>>
>> We’ve identified 14 test failures that must be resolved.  The issues you
>> are working on important for that, so thanks Chris.
>>
>>
>>
>> Here are the 14 and their current status:
>>
>>
>>
>> *Persistent-license failures - 5 Failures
>>               *
>>
>> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, destroy and acknowledge
>>
>> Pass on Chrome OS
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>> Pass on Chrome OS
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, persistent-license, mp4playback
>>
>> Pass on Chrome OS
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, retrieve, playback and
>> destroy
>>
>> Pass on Chrome OS
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, retrieve, playback
>>
>> Pass on Chrome OS
>>
>>
>>
>> *Persistent-usage-record failures - 3 Failures
>>                            *
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, check events
>>
>> No current implementation
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>
>> No current implementation
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4playback
>>
>> No current implementation
>>
>>
>>
>> *Temporary session failures - 4 Failures
>>               *
>>
>> /encrypted-media/drm-mp4-playback-temporary-expired.html
>>
>> drm, temporary, mp4, expired license
>>
>> Status TBD
>>
>>
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>> WPT issue 4027, test as written cannot be passed with Widevine
>>
>>
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>> sequential
>>
>> WPT pull 3985 expected to fix
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>> WPT pull 3985 expected to fix
>>
>>
>>
>> *Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures                                              *
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>> sequential, readyState
>>
>> EME issue #338 & #339 merged, should now pass on FF nightly
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>> EME issue #338 & #339 merged, should now pass on FF nightly
>>
>>
>>
>
> Note: Firefox Nightly won't pass the mp4-playback-temporary-multikey-sequential
> test assertions which count 'timeupdate' events until we resolve Mozilla
> *Bug 1312886* <https://bugzilla.mozilla.org/show_bug.cgi?id=1312886> in
> Firefox. The readyState behaviour is implemented, but we're not actually
> suspending playback when only one stream is blocked waiting for a key.
>
>
> Chris.
>
>
>
>
>> *From:* Chris Pearce [mailto:cpearce@mozilla.com]
>> *Sent:* Tuesday, October 25, 2016 3:06 PM
>> *To:* David Dorwin <ddorwin@google.com>
>> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Jerry Smith (WPT) <
>> jdsmith@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz
>> <wolenetz@google.com>; public-html-media@w3.org
>> *Subject:* Re: EME Test Status Update - 10-24-16
>>
>>
>>
>> OK, now that the above PR has landed, Firefox Nightly passes the
>> readyState check. We're still failing the "Should not continue playing
>> whilst waiting for a key expected" test, because it turns out that we don't
>> correctly suspend playback when waiting for key. Good thing we've got a web
>> platform test!
>>
>> Chris Pearce.
>>
>>
>>
>> On Wed, Oct 26, 2016 at 10:02 AM, David Dorwin <ddorwin@google.com>
>> wrote:
>>
>> https://github.com/w3c/web-platform-tests/pull/4076 fixes the readyState
>> expectation. If you're happy with it, we can merge and you can address the
>> playing count.
>>
>>
>>
>> On Tue, Oct 25, 2016 at 2:01 PM, Chris Pearce <cpearce@mozilla.com>
>> wrote:
>>
>> I am working on updating PR https://github.com/w3c/web-
>> platform-tests/pull/3985 now. This should make the .*sequential.* tests
>> pass.
>>
>>
>>
>> On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com> wrote:
>>
>> Some of these are likely due to a change in error type. See
>> https://github.com/w3c/web-platform-tests/issues/4029#
>> issuecomment-255476955 and https://github.com/w3c/web-
>> platform-tests/issues/4030#issuecomment-255499074. (Chrome now passes
>> the affected tests.)
>>
>>
>>
>> For the readyState tests, we have spec issue https://github.com/w3c/e
>> ncrypted-media/issues/336 and your open PR https://github.com/w3c/web-
>> platform-tests/pull/3985. I'm also preparing a PR to update the tests
>> for https://github.com/w3c/encrypted-media/issues/338. As discussed in
>> the telecon this morning, we are hoping that Firefox will pass the
>> (updated) readyState tests and demonstrate implementation since Chrome has
>> unrelated bugs.
>>
>>
>>
>> drm-temporary-license-type.html is incorrect and needs to be disabled or
>> fixed. See https://github.com/w3c/web-platform-tests/issues/4027#
>> issuecomment-255481770.
>>
>>
>>
>> On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com>
>> wrote:
>>
>> Currently the set of tests I see failing in a local Firefox build other
>> than persistent tests are these:
>>
>> /encrypted-media/clearkey-invalid-license.html    FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>> NOTRUN        0/1
>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html    FAIL
>> 6/7
>> /encrypted-media/clearkey-mp4-update-disallowed-input.html    FAIL
>>     0/1
>> /encrypted-media/clearkey-update-non-ascii-input.html    FAIL        0/1
>> /encrypted-media/drm-invalid-license.html    FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multisession.html
>> TIMEOUT        0/1
>> /encrypted-media/drm-mp4-syntax-mediakeysession.html    FAIL        6/7
>> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>> FAIL        0/1
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>> FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> FAIL        0/1
>>
>> Some of the tests have changed. We'll investigate.
>>
>> Chris.
>>
>>
>>
>>
>>
>> On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com>
>> wrote:
>>
>> David and Chris:
>>
>>
>>
>> Do these test results include the implementation changes you previously
>> announced in the following messages?
>>
>> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0044.html
>>
>> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0042.html
>>
>>
>>
>> /paulc
>>
>>
>>
>> *From:* Jerry Smith (WPT)
>> *Sent:* Monday, October 24, 2016 9:17 PM
>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>> wolenetz@google.com>; 'public-html-media@w3.org' <
>> public-html-media@w3.org>
>> *Subject:* EME Test Status Update - 10-24-16
>>
>>
>>
>> I’ve updated the test status reports to reflect recent fixes.  I believe
>> some improvement was expected from last Thursday, but the results are
>> largely the same.  I tested using:
>>
>>
>>
>> -         Chrome 56.0.2899.0 canary (64 bit)
>>
>> -         Firefox 52.0a1 (2016-10-24) (32 bit)
>>
>>
>>
>> Reports:
>>
>>
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>
>> o   *Test files*: 97; *Total subtests*: 306
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/complete-
>> fails.html
>>
>> o   *Completely failed files*: 21; *Completely failed subtests*: 21; *Failure
>> level*: 21/306 (6.86%)
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>> 2.html
>>
>> o   *Test files without 2 passes*: 33; *Subtests without 2 passes: *47; *Failure
>> level*: 47/306 (15.36%)
>>
>>
>>
>> Summary Groupings:
>>
>>
>>
>> Errors (considered bugs, not blocking)
>> 22
>>
>> Clear Key persistent-license (not planned, may make non-normative)       5
>>
>> Clear Key persistent-usage-record (not planned, at risk)                 3
>>
>> Clear Key transitions between clear and encrypted (bugs, not blocking)   2
>>
>>
>>
>> Non-Blocking issues                                                     3
>> 2
>>
>>
>>
>> Persistent-license (fixes in work)
>>             6
>>
>> Persistent-usage-record (no implementations in test, at risk)            3
>>
>> Temporary session edge cases (bugs, not block, fixes in work)            3
>>
>> Temporary session readyState (issues under discussion, at risk)          2
>>
>>
>>
>> In work or at risk
>> 14
>>
>>
>>
>> TOTAL
>> 46
>>
>>
>>
>> Details on failed tests are below.
>>
>>
>>
>> Jerry
>>
>>
>>
>>
>>
>> Error handling (considered bugs, not blocking) - 22 Failures
>>
>> /encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> Update with invalid Clear Key license
>>
>> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7, 14.29%,
>> 0.33% of total)
>>
>> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
>> various sessions
>>
>> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> update() with invalid response (longer than 64Kb characters) should fail.
>>
>> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> org.w3.clearkey test handling of non-ASCII responses for update()
>>
>> /encrypted-media/drm-generate-request-disallowed-input.html (3/7,
>> 42.86%, 0.98% of total)
>>
>> drm, temporary, keyids, initData longer than 64Kb characters
>>
>> drm, temporary, keyids, invalid initdata (too short key ID)
>>
>> drm, temporary, keyids, invalid initdata (too long key ID)
>>
>> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
>>
>> Update with invalid Clear Key license
>>
>> /encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%, 0.33%
>> of total)
>>
>> drm test if MediaKeySession generateRequest() resolves for various
>> sessions
>>
>> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>>
>> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
>> Object])
>>
>> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>>
>> MediaKeySystemAccess interface: operation createMediaKeys()
>>
>> MediaKeys interface: operation setServerCertificate(BufferSource)
>>
>> MediaKeySession interface: operation generateRequest(DOMString,Buff
>> erSource)
>>
>> MediaKeySession interface: operation load(DOMString)
>>
>> MediaKeySession interface: operation update(BufferSource)
>>
>> MediaKeySession interface: operation close()
>>
>> MediaKeySession interface: operation remove()
>>
>> MediaKeyMessageEvent interface object length
>>
>> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
>> 1.72%, 0.33% of total)
>>
>> Trailing space in contentType
>>
>> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%,
>> 0.65% of total)
>>
>> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4;
>> foo="bar"') should result in NotSupportedError
>>
>> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
>> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
>> NotSupportedError
>>
>>
>>
>> Clear key persistent-license tests (not planned to be implemented, may
>> make informative note) - 5 Failures
>>
>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>> acknowledge
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>> and destroy
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>>
>>
>>
>> Clear key persistent-usage-record (not planned to be implemented) - 3
>> Failures
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
>> window
>>
>>
>>
>> Clear key transitions between clear and encrypted content (considered
>> bugs, not blocking) - 2 Failures
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, temporary, mp4, playback, single key, clear then
>> encrypted content
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
>> clear content
>>
>>
>>
>> Persistent-license failures - 6 Failures
>>
>> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, destroy and acknowledge
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, persistent-license, mp4playback
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, retrieve, playback
>>
>>
>>
>> Persistent-usage-record failures - 3 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4playback
>>
>>
>>
>> Temporary session failures - 3 Failures
>>
>> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, temporary, mp4, expired license
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>> sequential
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>>
>>
>> Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures
>>
>> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
>> sequential, readyState
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>