W3C home > Mailing lists > Public > public-html-media@w3.org > October 2016

RE: EME Test Status Update - 10-27-16

From: Jerry Smith (WPT) <jdsmith@microsoft.com>
Date: Fri, 28 Oct 2016 00:28:20 +0000
To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Paul Cotton <Paul.Cotton@microsoft.com>, David Dorwin <ddorwin@google.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "'public-html-media@w3.org'" <public-html-media@w3.org>
Message-ID: <BY2PR03MB041B398E39D49B72287A4A6A4AD0@BY2PR03MB041.namprd03.prod.outlook.com>
Summary of issues related to each failure is:

Persistent-license failures - 1 Failures
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events

     This is being internally investigated at Google for a Chrome OS fix.

Temporary session failures - 2 Failures
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session

     This is being worked under https://github.com/w3c/web-platform-tests/issues/4027 assigned to mwatson2<https://github.com/w3c/web-platform-tests/issues/4027%20assigned%20to%20mwatson2>

/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, playback, check events

     This is actually a new failure that didn't occur on the 10/24/2016 pass.  Can anyone comment on the test?

Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential

     This issue is dependent on some spec clarifications in these issues:
     https://github.com/w3c/encrypted-media/issues/338 - closed
     https://github.com/w3c/encrypted-media/issues/339 - closed
     https://github.com/w3c/encrypted-media/issues/336 - in review

     I'm not sure if we have an issue opened to modify the test once the spec issues are cleared.

Jerry

From: Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
Sent: Thursday, October 27, 2016 4:40 PM
To: Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; 'public-html-media@w3.org' <public-html-media@w3.org>
Subject: EME Test Status Update - 10-27-16

I've posted updated test status reports to reflect a few bug fixes and also to add Chrome OS results.

Reports:


*        http://w3c.github.io/test-results/encrypted-media/all.html

o   Test files: 97; Total subtests: 306

*        http://w3c.github.io/test-results/encrypted-media/complete-fails.html

o   Completely failed files: 16; Completely failed subtests: 16; Failure level: 16/306 (5.23%)

*        http://w3c.github.io/test-results/encrypted-media/less-than-2.html

o   Test files without 2 passes: 28; Subtests without 2 passes: 40; Failure level: 40/306 (13.07%)
Comments:


-   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from 47 to 40 subtest failures.

-   Persistent-license failures dropped from 5 to 1 with the addition of Chrome on Chrome OS results.

-   One temporary session test failed that was previously passing:

o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)

o   drm, temporary, mp4, playback, check events

-   There are only 5 tests in complete-fails that we are still working on for our conformance record.

-   That reflects our revised criteria of having a single passing implementation (in addition to broad interoperable experience).
In line with that criteria, I've chosen to break down just the complete-fails test results for this summary.

Browser Versions:

CD56:  Chrome 56.0.2902.0 canary (64-bit)
CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)

Summary Groupings;

Errors (considered bugs, not blocking)                                     0
Clear Key persistent-license (not planned, may make non-normative)         5
Clear Key persistent-usage-record (at risk, planned to be removed)         3
Clear Key transitions between clear and encrypted (bugs, not blocking)     0

Not Blocking Tests                                                         8

Persistent-license (fixes in work)                                         1
Persistent-usage-record (at risk, planned to be removed)                   3
Temporary session edge cases (bugs, not block, fixes in work)              2
Temporary session readyState (issues under discussion, at risk)            2

In work or at risk                                                         8

TOTAL                                                                     16


Details on failed tests are below.

Jerry


Error handling (considered bugs, not blocking) - 0 Failures

Clear key persistent-license tests (not planned to be implemented, may make informative note) - 5 Failures
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback

Clear key persistent-usage-record (not planned to be implemented) - 3 Failures
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window

Clear key transitions between clear and encrypted content (considered bugs, not blocking) - 2 Failures

Persistent-license failures - 6 Failures
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events

Persistent-usage-record failures - 3 Failures
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, check events
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, retrieve in new window
/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4playback

Temporary session failures - 2 Failures
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, playback, check events

Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential
Received on Friday, 28 October 2016 00:29:01 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 15:49:17 UTC