- From: Chris Pearce <cpearce@mozilla.com>
- Date: Wed, 26 Oct 2016 13:48:06 +1300
- To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>
- Cc: David Dorwin <ddorwin@google.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "public-html-media@w3.org" <public-html-media@w3.org>
- Message-ID: <CAFWdrwa7oBDv3Y-XYe1RjrcEsAdK6F9p7tiJQECFFHjjim0cfA@mail.gmail.com>
On Wed, Oct 26, 2016 at 1:20 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:
> I don’t see /encrypted-media/drm-mp4-playback-temporary-expired listed
> below. Is it on your radar, or passing in your Nightly?
>
encrypted-media/drm-mp4-playback-temporary-expired
<http://w3c-test.org/encrypted-media/drm-mp4-playback-temporary-expired.html>
is intermittently passing about half the time in Firefox Nightly on my
local machine. We're looking into this one still.
>
> We’ve identified 14 test failures that must be resolved. The issues you
> are working on important for that, so thanks Chris.
>
>
>
> Here are the 14 and their current status:
>
>
>
> *Persistent-license failures - 5 Failures
> *
>
> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, destroy and acknowledge
>
> Pass on Chrome OS
>
>
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
> Pass on Chrome OS
>
>
>
> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, persistent-license, mp4playback
>
> Pass on Chrome OS
>
>
>
> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> Pass on Chrome OS
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback
>
> Pass on Chrome OS
>
>
>
> *Persistent-usage-record failures - 3 Failures
> *
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, check events
>
> No current implementation
>
>
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> No current implementation
>
>
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4playback
>
> No current implementation
>
>
>
> *Temporary session failures - 4 Failures
> *
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html
>
> drm, temporary, mp4, expired license
>
> Status TBD
>
>
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
> WPT issue 4027, test as written cannot be passed with Widevine
>
>
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential
>
> WPT pull 3985 expected to fix
>
>
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
> WPT pull 3985 expected to fix
>
>
>
> *Temporary session readyState failures (listed at risk, under discussion)
> - 2 Failures *
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential, readyState
>
> EME issue #338 & #339 merged, should now pass on FF nightly
>
>
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> EME issue #338 & #339 merged, should now pass on FF nightly
>
>
>
Note: Firefox Nightly won't pass the
mp4-playback-temporary-multikey-sequential test assertions which count
'timeupdate' events until we resolve Mozilla *Bug 1312886*
<https://bugzilla.mozilla.org/show_bug.cgi?id=1312886> in Firefox. The
readyState behaviour is implemented, but we're not actually suspending
playback when only one stream is blocked waiting for a key.
Chris.
> *From:* Chris Pearce [mailto:cpearce@mozilla.com]
> *Sent:* Tuesday, October 25, 2016 3:06 PM
> *To:* David Dorwin <ddorwin@google.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Jerry Smith (WPT) <
> jdsmith@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
> wolenetz@google.com>; public-html-media@w3.org
> *Subject:* Re: EME Test Status Update - 10-24-16
>
>
>
> OK, now that the above PR has landed, Firefox Nightly passes the
> readyState check. We're still failing the "Should not continue playing
> whilst waiting for a key expected" test, because it turns out that we don't
> correctly suspend playback when waiting for key. Good thing we've got a web
> platform test!
>
> Chris Pearce.
>
>
>
> On Wed, Oct 26, 2016 at 10:02 AM, David Dorwin <ddorwin@google.com> wrote:
>
> https://github.com/w3c/web-platform-tests/pull/4076 fixes the readyState
> expectation. If you're happy with it, we can merge and you can address the
> playing count.
>
>
>
> On Tue, Oct 25, 2016 at 2:01 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
> I am working on updating PR https://github.com/w3c/web-
> platform-tests/pull/3985 now. This should make the .*sequential.* tests
> pass.
>
>
>
> On Wed, Oct 26, 2016 at 9:45 AM, David Dorwin <ddorwin@google.com> wrote:
>
> Some of these are likely due to a change in error type. See
> https://github.com/w3c/web-platform-tests/issues/
> 4029#issuecomment-255476955 and https://github.com/w3c/
> web-platform-tests/issues/4030#issuecomment-255499074. (Chrome now passes
> the affected tests.)
>
>
>
> For the readyState tests, we have spec issue https://github.com/w3c/
> encrypted-media/issues/336 and your open PR https://github.com/w3c/web-
> platform-tests/pull/3985. I'm also preparing a PR to update the tests for
> https://github.com/w3c/encrypted-media/issues/338. As discussed in the
> telecon this morning, we are hoping that Firefox will pass the (updated)
> readyState tests and demonstrate implementation since Chrome has unrelated
> bugs.
>
>
>
> drm-temporary-license-type.html is incorrect and needs to be disabled or
> fixed. See https://github.com/w3c/web-platform-tests/issues/
> 4027#issuecomment-255481770.
>
>
>
> On Tue, Oct 25, 2016 at 1:37 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
> Currently the set of tests I see failing in a local Firefox build other
> than persistent tests are these:
>
> /encrypted-media/clearkey-invalid-license.html FAIL 0/1
> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
> NOTRUN 0/1
> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html FAIL
> 6/7
> /encrypted-media/clearkey-mp4-update-disallowed-input.html FAIL
> 0/1
> /encrypted-media/clearkey-update-non-ascii-input.html FAIL 0/1
> /encrypted-media/drm-invalid-license.html FAIL 0/1
> /encrypted-media/drm-mp4-playback-temporary-multisession.html
> TIMEOUT 0/1
> /encrypted-media/drm-mp4-syntax-mediakeysession.html FAIL 6/7
> /encrypted-media/drm-temporary-license-type.html FAIL 0/1
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
> FAIL 0/1
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
> FAIL 0/1
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> FAIL 0/1
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> FAIL 0/1
>
> Some of the tests have changed. We'll investigate.
>
> Chris.
>
>
>
>
>
> On Tue, Oct 25, 2016 at 3:30 PM, Paul Cotton <Paul.Cotton@microsoft.com>
> wrote:
>
> David and Chris:
>
>
>
> Do these test results include the implementation changes you previously
> announced in the following messages?
>
> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0044.html
>
> https://lists.w3.org/Archives/Public/public-html-media/2016Oct/0042.html
>
>
>
> /paulc
>
>
>
> *From:* Jerry Smith (WPT)
> *Sent:* Monday, October 24, 2016 9:17 PM
> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
> wolenetz@google.com>; 'public-html-media@w3.org' <public-html-media@w3.org
> >
> *Subject:* EME Test Status Update - 10-24-16
>
>
>
> I’ve updated the test status reports to reflect recent fixes. I believe
> some improvement was expected from last Thursday, but the results are
> largely the same. I tested using:
>
>
>
> - Chrome 56.0.2899.0 canary (64 bit)
>
> - Firefox 52.0a1 (2016-10-24) (32 bit)
>
>
>
> Reports:
>
>
>
> · http://w3c.github.io/test-results/encrypted-media/all.html
>
> o *Test files*: 97; *Total subtests*: 306
>
> · http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> o *Completely failed files*: 21; *Completely failed subtests*: 21; *Failure
> level*: 21/306 (6.86%)
>
> · http://w3c.github.io/test-results/encrypted-media/less-
> than-2.html
>
> o *Test files without 2 passes*: 33; *Subtests without 2 passes: *47; *Failure
> level*: 47/306 (15.36%)
>
>
>
> Summary Groupings:
>
>
>
> Errors (considered bugs, not blocking) 22
>
> Clear Key persistent-license (not planned, may make non-normative) 5
>
> Clear Key persistent-usage-record (not planned, at risk) 3
>
> Clear Key transitions between clear and encrypted (bugs, not blocking) 2
>
>
>
> Non-Blocking issues 32
>
>
>
> Persistent-license (fixes in work) 6
>
> Persistent-usage-record (no implementations in test, at risk) 3
>
> Temporary session edge cases (bugs, not block, fixes in work) 3
>
> Temporary session readyState (issues under discussion, at risk) 2
>
>
>
> In work or at risk 14
>
>
>
> TOTAL 46
>
>
>
> Details on failed tests are below.
>
>
>
> Jerry
>
>
>
>
>
> Error handling (considered bugs, not blocking) - 22 Failures
>
> /encrypted-media/clearkey-invalid-license.html (1/1, 100.00%, 0.33% of
> total)
>
> Update with invalid Clear Key license
>
> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (1/7, 14.29%,
> 0.33% of total)
>
> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
> various sessions
>
> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%,
> 0.33% of total)
>
> update() with invalid response (longer than 64Kb characters) should fail.
>
> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
> 0.33% of total)
>
> org.w3.clearkey test handling of non-ASCII responses for update()
>
> /encrypted-media/drm-generate-request-disallowed-input.html (3/7, 42.86%,
> 0.98% of total)
>
> drm, temporary, keyids, initData longer than 64Kb characters
>
> drm, temporary, keyids, invalid initdata (too short key ID)
>
> drm, temporary, keyids, invalid initdata (too long key ID)
>
> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
>
> Update with invalid Clear Key license
>
> /encrypted-media/drm-mp4-syntax-mediakeysession.html (1/7, 14.29%, 0.33%
> of total)
>
> drm test if MediaKeySession generateRequest() resolves for various sessions
>
> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>
> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
> Object])
>
> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>
> MediaKeySystemAccess interface: operation createMediaKeys()
>
> MediaKeys interface: operation setServerCertificate(BufferSource)
>
> MediaKeySession interface: operation generateRequest(DOMString,
> BufferSource)
>
> MediaKeySession interface: operation load(DOMString)
>
> MediaKeySession interface: operation update(BufferSource)
>
> MediaKeySession interface: operation close()
>
> MediaKeySession interface: operation remove()
>
> MediaKeyMessageEvent interface object length
>
> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
> 1.72%, 0.33% of total)
>
> Trailing space in contentType
>
> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%,
> 0.65% of total)
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4;
> foo="bar"') should result in NotSupportedError
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
> NotSupportedError
>
>
>
> Clear key persistent-license tests (not planned to be implemented, may
> make informative note) - 5 Failures
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>
>
>
> Clear key persistent-usage-record (not planned to be implemented) - 3
> Failures
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
> window
>
>
>
> Clear key transitions between clear and encrypted content (considered
> bugs, not blocking) - 2 Failures
>
> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, temporary, mp4, playback, single key, clear then
> encrypted content
>
> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
> clear content
>
>
>
> Persistent-license failures - 6 Failures
>
> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, destroy and acknowledge
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, persistent-license, mp4playback
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback
>
>
>
> Persistent-usage-record failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4playback
>
>
>
> Temporary session failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, expired license
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential, readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
>
>
>
>
>
>
>
>
>
>
Received on Wednesday, 26 October 2016 00:48:45 UTC