- From: Chris Pearce <cpearce@mozilla.com>
- Date: Fri, 21 Oct 2016 16:17:15 +1300
- To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>
- Cc: Paul Cotton <Paul.Cotton@microsoft.com>, "public-html-media@w3.org" <public-html-media@w3.org>, David Dorwin <ddorwin@google.com>, Mark Watson <watsonm@netflix.com>
- Message-ID: <CAFWdrwarN+_wOF41UEhKiMgb_rQuf-WFNgmPdcM65fm9XAbL3g@mail.gmail.com>
I have an update on Firefox's progress. In my local Firefox build which
contains a few patches that I have just pushed into our integration
pipeline, we now pass most of the tests at current master of the
web-platform-tests Github repo.
(The online test runner at w3c-test.org
<http://w3c-test.org/tools/runner/index.html> seems to run a different set
of tests, I assume it's out of date and the GitHub repo should be
considered the source of truth for tests?)
We now fail the following tests:
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-persistent-license.html FAIL
0/1
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
FAIL 0/1
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
FAIL 0/1
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html
FAIL 0/1
/encrypted-media/drm-mp4-playback-persistent-license-events.html FAIL
0/1
/encrypted-media/drm-mp4-playback-persistent-license.html FAIL 0/1
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
FAIL 0/1
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
FAIL 0/1
*/encrypted-media/drm-mp4-playback-temporary-expired.html FAIL
0/1/encrypted-media/drm-mp4-playback-temporary-multisession.html FAIL
0/1/encrypted-media/drm-temporary-license-type.html FAIL
0/1/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
FAIL
0/1/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.html
FAIL 0/1*/encrypted-media/drm-mp4-playback-persistent-usage-record.html
FAIL 0/1
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
FAIL 0/1
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html
FAIL 0/1
*/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
FAIL
0/1/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
FAIL 0/1*
The tests in non-bold are persistent session tests, and we're not going to
make them pass.
Of the remainder;
*drm-mp4-playback-temporary-expired.html* is a problem with DRMToday, WPT
issue 4026 <https://github.com/w3c/web-platform-tests/issues/4026>.
*drm-mp4-playback-temporary-multisession.html* fails on first run only or
if not visible; reruns while visible pass. I expect this is a bug in our
code. I don't think we need to block the spec on this.
*drm-temporary-license-type.html* - a problem with DRMToday, WPT issue 4027
<https://github.com/w3c/web-platform-tests/issues/4027>.
The next two tests;
*clearkey-mp4-playback-temporary-multikey-sequential.html* and
*drm-mp4-playback-temporary-multikey-sequential.html*
... fail because we've not merged PR 3985
<https://github.com/w3c/web-platform-tests/pull/3985>;
The remaining two tests;
*clearkey-mp4-playback-temporary-multikey-sequential-readyState.html * and
*drm-mp4-playback-temporary-multikey-sequential-readyState.html*
... also fail because we've not merged PR 3985
<https://github.com/w3c/web-platform-tests/pull/3985>, and additionally
Firefox is at readyState=HAVE_CURRENT_DATA when we dispatch the
waitingforkey event, and this test is checking that we're at
readyState=HAVE_METADATA when we fire waitingforkey.
The current draft spec says we should be at readyState=HAVE_METADATA when
we fire "waitingforkey", but Mozilla's media playback teak thinks that it
makes more sense for us to be at readyState=HAVE_CURRENT_DATA, as the last
frame we decrypted and decoded is at the current playback position and is
currently being rendered. This is discussed in EME issue 338
<https://github.com/w3c/encrypted-media/issues/338> and issue 339
<https://github.com/w3c/encrypted-media/issues/339>. We'd like to see the
EME spec change to not require readyState to drop back to HAVE_METADATA
when firing "waitingforkey", and our implementation reflects that. PR 3985
<https://github.com/w3c/web-platform-tests/pull/3985> will need to change
once issue 338 <https://github.com/w3c/encrypted-media/issues/338> is
resolved.
I expect that the fixes in my local build should make it into Firefox
Nightly 52 before Monday morning Pacific Time. So Jerry should be able to
update Firefox Nightly on Monday morning and retest then.
Regards,
Chris Pearce.
On Thu, Oct 20, 2016 at 10:29 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:
> I’ve updated the test status reports to reflect recent fixes. More are
> expected in the next few days.
>
>
>
> · http://w3c.github.io/test-results/encrypted-media/all.html
>
> o *Test files*: 97; *Total subtests*: 306
>
> · http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> o *Completely failed files*: 22; *Completely failed subtests*: 22; *Failure
> level*: 22/306 (7.19%)
>
> · http://w3c.github.io/test-results/encrypted-media/less-than-2.html
>
> o *Test files without 2 passes*: 33; *Subtests without 2 passes: *47; *Failure
> level*: 47/306 (15.36%)
>
> We discussed in the call on Tuesday treating some categories of test
> issues as non-blocking bugs. There are in fact few issues currently that
> are viewed as feature failures on our tests. I did a sort through these
> and am slightly off on the total tests. I am out tomorrow, so am posting
> this as an interim status for comments, despite this.
>
>
>
> In the category we consider non-blocking bugs are these:
>
>
>
> Errors (considered bugs, not blocking)
> 23
>
> Clear Key persistent-license (not planned, may make non-normative)
> 3
>
> Clear Key persistent-usage-record (not planned, no impact)
> 3
>
> Clear Key transitions between clear and encrypted (bugs, not
> blocking) 2
>
>
>
> Not Blocking Tests Total:
> 31
>
>
>
> In the category we are treating as implementation issues currently are
> these:
>
>
>
> Persistent-license (fixes in work)
> 5
>
> Persistent-usage-record (no implementations in test, at risk)
> 3
>
> Temporary session edge cases (bugs, not block, fixes in work)
> 3
>
> Temporary session readyState (issues under discussion, at risk)
> 2
>
>
>
> In work or at risk Total:
> 13
>
>
>
> Lists of the individual tests sorted into these groups are provided
> below. Comments on the groupings are welcome.
>
>
>
> Jerry
>
>
>
>
>
> Error handling (considered bugs, not blocking) - 23 Failures
>
> /encrypted-media/clearkey-mp4-syntax-mediakeysession.html (2/7, 28.57%,
> 0.65% of total)
>
> org.w3.clearkey test MediaKeySession load() exceptions.
>
> org.w3.clearkey test if MediaKeySession generateRequest() resolves for
> various sessions
>
> /encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%,
> 0.33% of total)
>
> org.w3.clearkey test handling of non-ASCII responses for update()
>
> /encrypted-media/drm-generate-request-disallowed-input.html (3/7, 42.86%,
> 0.98% of total)
>
> drm, temporary, keyids, initData longer than 64Kb characters
>
> drm, temporary, keyids, invalid initdata (too short key ID)
>
> drm, temporary, keyids, invalid initdata (too long key ID)
>
> /encrypted-media/drm-mp4-syntax-mediakeysession.html (2/7, 28.57%, 0.65%
> of total)
>
> drm test MediaKeySession load() exceptions.
>
> drm test if MediaKeySession generateRequest() resolves for various sessions
>
> /encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
>
> Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
> Object])
>
> HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
>
> MediaKeySystemAccess interface: operation createMediaKeys()
>
> MediaKeys interface: operation setServerCertificate(BufferSource)
>
> MediaKeySession interface: operation generateRequest(DOMString,
> BufferSource)
>
> MediaKeySession interface: operation load(DOMString)
>
> MediaKeySession interface: operation update(BufferSource)
>
> MediaKeySession interface: operation close()
>
> MediaKeySession interface: operation remove()
>
> MediaKeyMessageEvent interface object length
>
> /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58,
> 1.72%, 0.33% of total)
>
> Trailing space in contentType
>
> /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%,
> 0.65% of total)
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4;
> foo="bar"') should result in NotSupportedError
>
> drm, requestMediaKeySystemAccess: Unrecognized foo with codecs
> ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in
> NotSupportedError
>
> /encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%,
> 0.33% of total)
>
> update() with invalid response (longer than 64Kb characters) should fail.
>
> /encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
>
> Update with invalid Clear Key license
>
>
>
> Clear key persistent license tests (not planned to be implemented, may
> make informative note) - 3 Failures
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>
>
>
> Clear key persistent-usage-case (not planned to be implemented) - 3
> Failures
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
> window
>
>
>
> Clear key transitions between clear and encrypted content (considered
> bugs, not blocking) - 2 Failures
>
> /encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, temporary, mp4, playback, single key, clear then
> encrypted content
>
> /encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, temporary, mp4, playback, single key, encrypted then
> clear content
>
>
>
> Persistent-license failures - 5 Failures
>
> /encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, destroy and acknowledge
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, persistent-license, mp4playback
>
> /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback and destroy
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, retrieve, playback
>
>
>
> Persistent-usage-record failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4playback
>
>
>
> Temporary session failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, expired license
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, successful playback, temporary, mp4, multiple keys,
> sequential, readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
Received on Friday, 21 October 2016 03:17:46 UTC