Re: EME Test Status Update - 10-27-16

Thanks, Jerry.

The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html
failure is tracked in https://crbug.com/659828.

The drm-mp4-playback-persistent-license-events.html failure on Chrome OS is
tracked as https://crbug.com/660393.

drm-mp4-playback-temporary-expired.html is flaky due to timing. I just saw
a failure and a pass.

drm-mp4-unique-origin.html appears to be a new failure. We're looking into
it.

On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com> wrote:

> The failure in /encrypted-media/drm-mp4-playback-temporary-events.html
> (at least in Chrome and in Firefox) is caused by Widevine CDM resolving
> MediaKeySession.closed and the promise returned by MediaKeySession.close()
> in the wrong order.
>
> The test changed with issue 4058
> <https://github.com/w3c/web-platform-tests/issues/4058> to expect that
> the MediaKeySession.closed promise is resolved before the promise returned
> by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on
> the Widevine CDM to resolve the promise in the correct order, and it
> doesn't. So this is an implementation bug in the Widevine CDM.
>
> The ClearKey variant of this test
> <https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
> should be fixed in Firefox Nightly in a day or so, and is already fixed in
> the latest Chrome Canary.
>
> We have patches in the works for fixing drm-mp4-playback-temporary-
> multikey-sequential[-readyState].html.
>
> Chris.
>
>
>
> On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
>> Summary of issues related to each failure is:
>>
>>
>>
>> Persistent-license failures - 1 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>>
>>
>>      This is being internally investigated at Google for a Chrome OS fix.
>>
>>
>>
>> Temporary session failures - 2 Failures
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>>
>>
>>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
>> assigned to mwatson2
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, temporary, mp4, playback, check events
>>
>>
>>
>>      This is actually a new failure that didn’t occur on the 10/24/2016
>> pass.  Can anyone comment on the test?
>>
>>
>>
>> Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>>
>>
>>      This issue is dependent on some spec clarifications in these issues:
>>
>>      https://github.com/w3c/encrypted-media/issues/338 - closed
>>
>>      https://github.com/w3c/encrypted-media/issues/339 - closed
>>
>>      https://github.com/w3c/encrypted-media/issues/336 - in review
>>
>>
>>
>>      I’m not sure if we have an issue opened to modify the test once the
>> spec issues are cleared.
>>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
>> *Sent:* Thursday, October 27, 2016 4:40 PM
>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>> wolenetz@google.com>; 'public-html-media@w3.org' <
>> public-html-media@w3.org>
>> *Subject:* EME Test Status Update - 10-27-16
>>
>>
>>
>> I’ve posted updated test status reports to reflect a few bug fixes and
>> also to add Chrome OS results.
>>
>>
>>
>> Reports:
>>
>>
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>
>> o   *Test files: 97; Total subtests: 306*
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/complete-
>> fails.html
>>
>> o   *Completely failed files: 16; Completely failed subtests: 16;
>> Failure level: 16/306 (5.23%)*
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>> 2.html
>>
>> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
>> Failure level: 40/306 (13.07%)*
>>
>> Comments:
>>
>>
>>
>> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from
>> 47 to 40 subtest failures.
>>
>> -   Persistent-license failures dropped from 5 to 1 with the addition of
>> Chrome on Chrome OS results.
>>
>> -   One temporary session test failed that was previously passing:
>>
>> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> o   drm, temporary, mp4, playback, check events
>>
>> -   There are only 5 tests in complete-fails that we are still working
>> on for our conformance record.
>>
>> -   That reflects our revised criteria of having a single passing
>> implementation (in addition to broad interoperable experience).
>>
>> In line with that criteria, I’ve chosen to break down just the
>> complete-fails test results for this summary.
>>
>>
>>
>> Browser Versions:
>>
>>
>>
>> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>>
>> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>>
>> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>>
>> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>>
>>
>>
>> Summary Groupings;
>>
>>
>>
>> Errors (considered bugs, not blocking)
>> 0
>>
>> Clear Key persistent-license (not planned, may make
>> non-normative)         5
>>
>> Clear Key persistent-usage-record (at risk, planned to be
>> removed)         3
>>
>> Clear Key transitions between clear and encrypted (bugs, not
>> blocking)     0
>>
>>
>>
>> Not Blocking Tests
>>  8
>>
>>
>>
>> Persistent-license (fixes in work)
>>           1
>>
>> Persistent-usage-record (at risk, planned to be
>> removed)                   3
>>
>> Temporary session edge cases (bugs, not block, fixes in
>> work)              2
>>
>> Temporary session readyState (issues under discussion, at risk)
>>  2
>>
>>
>>
>> In work or at risk
>> 8
>>
>>
>>
>> TOTAL
>> 16
>>
>>
>>
>>
>>
>> Details on failed tests are below.
>>
>>
>>
>> Jerry
>>
>>
>>
>>
>>
>> Error handling (considered bugs, not blocking) - 0 Failures
>>
>>
>>
>> Clear key persistent-license tests (not planned to be implemented, may
>> make informative note) - 5 Failures
>>
>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>> acknowledge
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>> and destroy
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>>
>>
>>
>> Clear key persistent-usage-record (not planned to be implemented) - 3
>> Failures
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
>> window
>>
>>
>>
>> Clear key transitions between clear and encrypted content (considered
>> bugs, not blocking) - 2 Failures
>>
>>
>>
>> Persistent-license failures - 6 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>>
>>
>> Persistent-usage-record failures - 3 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4playback
>>
>>
>>
>> Temporary session failures - 2 Failures
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, temporary, mp4, playback, check events
>>
>>
>>
>> Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>
>

Received on Friday, 28 October 2016 16:10:55 UTC