Re: EME Test Status Update - 10-27-16

On Mon, Oct 31, 2016 at 4:24 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> We seem very close to finishing.  Do we have a shot at closing open test
> issues this week?
>
>
>
> David;  Can you clarify if you consider the new issues you’ve listed as
> V1?  I think you’ve added these two to the list:
>
>
>
> -          drm-mp4-playback-temporary-expired.html
>
> o   Flaky due to timing.
>
This may be a limitation of what we can do in a short test that depends on
an external server. Someone could take a closer look, but it may just be
that we need to rerun this if we see failures on Chrome or Firefox.

> -          drm-mp4-unique-origin.html
>
> o   New failure
>
Chris fixed this test, which was broken by a testharness update, but there
is now a question of whether we should disable/delete it.

>
>
> My previous list:
>
>
>
> -          drm-mp4-playback-persistent-license-events.html
>
> o   https://crbug.com/660393
>
> o   ETA?
>
This will require pushing a new CDM. We should consider this an
implementation bug with commitment to fix.

> -          drm-temporary-license-type.html
>
>                             Widevine CDM returns a temporary license when
> a persistent one is requested.
>
>                             Proposal in https://github.com/w3c/web-
> platform-tests/issues/4027 to rework the test.
>
>                             Is this must fix for V1?
>

Unless someone volunteers to implement Mark's proposal, we should probably
disable it for now.

> -          drm-mp4-playback-temporary-events.html
>
> o   https://crbug.com/659828
>
> o   ETA?
>
This will require pushing a new CDM. We should consider this an
implementation bug with commitment to fix. Chrome and Firefox, I believe,
have already fixed Clear Key.

> -          drm-mp4-playback-temporary-multikey-sequential-readyState.html
>
Chris was working on readyState in Firefox, so hopefully he can give an
update.

Chrome actually implements readyState but fails this test due to
https://crbug.com/658422, which is caused by a non-EME implementation
decision. We intend to fix the EME side but do not have an ETA.
https://crbug.com/658422#c2 is a patch that demonstrates this difference.

> -          drm-mp4-playback-temporary-multikey-sequential.html
>
> o   Am I correct that multiple PRs are targeted at this issue?
>
> o   Chris Pearce reports fixes are in work.
>
> o   What’s the prospects for closing this week?
>
The spec and test were changed to expect HAVE_CURRENT_DATA. Chris, what is
the status of Firefox?

Chrome fails for the same reason as the previous test.

>
>
> I added a V1 milestone to the test repository and marked one issue V1 (
> https://github.com/w3c/web-platform-tests/issues/4027) as V1.  There are
> 10 other test issues without a milestone, and 5 marked VNext.  Can folks
> weigh in on which should be treated as V1?
>

I punted a bunch of request for new tests.  The remaining four issues (
https://github.com/w3c/web-platform-tests/issues?q=is%3Aopen+label%3Aencrypted-media+no%3Amilestone
includes PRs) are minor issues we could try to fix but aren't required,
have a PR out for, or might have a test for, so I didn't punt them.

>
>
> Jerry
>
>
>
> *From:* David Dorwin [mailto:ddorwin@google.com]
> *Sent:* Friday, October 28, 2016 9:10 AM
> *To:* Chris Pearce <cpearce@mozilla.com>
> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <
> Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt
> Wolenetz <wolenetz@google.com>; public-html-media@w3.org
> *Subject:* Re: EME Test Status Update - 10-27-16
>
>
>
> Thanks, Jerry.
>
>
>
> The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html
> failure is tracked in https://crbug.com/659828.
>
>
>
> The drm-mp4-playback-persistent-license-events.html failure on Chrome OS
> is tracked as https://crbug.com/660393.
>
>
>
> drm-mp4-playback-temporary-expired.html is flaky due to timing. I just
> saw a failure and a pass.
>
>
>
> drm-mp4-unique-origin.html appears to be a new failure. We're looking into
> it.
>
>
>
> On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
> The failure in /encrypted-media/drm-mp4-playback-temporary-events.html
> (at least in Chrome and in Firefox) is caused by Widevine CDM resolving
> MediaKeySession.closed and the promise returned by MediaKeySession.close()
> in the wrong order.
>
> The test changed with issue 4058
> <https://github.com/w3c/web-platform-tests/issues/4058> to expect that
> the MediaKeySession.closed promise is resolved before the promise returned
> by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on
> the Widevine CDM to resolve the promise in the correct order, and it
> doesn't. So this is an implementation bug in the Widevine CDM.
>
> The ClearKey variant of this test
> <https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
> should be fixed in Firefox Nightly in a day or so, and is already fixed in
> the latest Chrome Canary.
>
>
>
> We have patches in the works for fixing drm-mp4-playback-temporary-
> multikey-sequential[-readyState].html.
>
>
>
> Chris.
>
>
>
>
>
> On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> Summary of issues related to each failure is:
>
>
>
> Persistent-license failures - 1 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
>
>
>      This is being internally investigated at Google for a Chrome OS fix.
>
>
>
> Temporary session failures - 2 Failures
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
>
>
>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
> assigned to mwatson2
>
>
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, playback, check events
>
>
>
>      This is actually a new failure that didn’t occur on the 10/24/2016
> pass.  Can anyone comment on the test?
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
>
>
>      This issue is dependent on some spec clarifications in these issues:
>
>      https://github.com/w3c/encrypted-media/issues/338 - closed
>
>      https://github.com/w3c/encrypted-media/issues/339 - closed
>
>      https://github.com/w3c/encrypted-media/issues/336 - in review
>
>
>
>      I’m not sure if we have an issue opened to modify the test once the
> spec issues are cleared.
>
>
>
> Jerry
>
>
>
> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
> *Sent:* Thursday, October 27, 2016 4:40 PM
> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
> wolenetz@google.com>; 'public-html-media@w3.org' <public-html-media@w3.org
> >
> *Subject:* EME Test Status Update - 10-27-16
>
>
>
> I’ve posted updated test status reports to reflect a few bug fixes and
> also to add Chrome OS results.
>
>
>
> Reports:
>
>
>
> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>
> o   *Test files: 97; Total subtests: 306*
>
> ·        http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> o   *Completely failed files: 16; Completely failed subtests: 16; Failure
> level: 16/306 (5.23%)*
>
> ·        http://w3c.github.io/test-results/encrypted-media/less-
> than-2.html
>
> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
> Failure level: 40/306 (13.07%)*
>
> Comments:
>
>
>
> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from 47
> to 40 subtest failures.
>
> -   Persistent-license failures dropped from 5 to 1 with the addition of
> Chrome on Chrome OS results.
>
> -   One temporary session test failed that was previously passing:
>
> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
> 100.00%, 0.33% of total)
>
> o   drm, temporary, mp4, playback, check events
>
> -   There are only 5 tests in complete-fails that we are still working on
> for our conformance record.
>
> -   That reflects our revised criteria of having a single passing
> implementation (in addition to broad interoperable experience).
>
> In line with that criteria, I’ve chosen to break down just the
> complete-fails test results for this summary.
>
>
>
> Browser Versions:
>
>
>
> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>
> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>
> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>
> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>
>
>
> Summary Groupings;
>
>
>
> Errors (considered bugs, not blocking)
> 0
>
> Clear Key persistent-license (not planned, may make non-normative)
> 5
>
> Clear Key persistent-usage-record (at risk, planned to be removed)
> 3
>
> Clear Key transitions between clear and encrypted (bugs, not blocking)
> 0
>
>
>
> Not Blocking Tests
>  8
>
>
>
> Persistent-license (fixes in work)
>           1
>
> Persistent-usage-record (at risk, planned to be removed)
> 3
>
> Temporary session edge cases (bugs, not block, fixes in work)
> 2
>
> Temporary session readyState (issues under discussion, at risk)
>  2
>
>
>
> In work or at risk
> 8
>
>
>
> TOTAL
> 16
>
>
>
>
>
> Details on failed tests are below.
>
>
>
> Jerry
>
>
>
>
>
> Error handling (considered bugs, not blocking) - 0 Failures
>
>
>
> Clear key persistent-license tests (not planned to be implemented, may
> make informative note) - 5 Failures
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and
> destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>
>
>
> Clear key persistent-usage-record (not planned to be implemented) - 3
> Failures
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
> window
>
>
>
> Clear key transitions between clear and encrypted content (considered
> bugs, not blocking) - 2 Failures
>
>
>
> Persistent-license failures - 6 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-license, mp4, playback, check events
>
>
>
> Persistent-usage-record failures - 3 Failures
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4, playback, retrieve in new window
>
> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.33% of total)
>
> drm, persistent-usage-record, mp4playback
>
>
>
> Temporary session failures - 2 Failures
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
> drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)
>
> drm, temporary, mp4, playback, check events
>
>
>
> Temporary session readyState failures (listed at risk, under discussion) -
> 2 Failures
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential,
> readyState
>
> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.33% of total)
>
> drm, successful playback, temporary, mp4, multiple keys, sequential
>
>
>
>
>