EME Test Status Update - 11/02/16

I’ve posted updated test status reports to EME, with two notable outcomes:
·       Only 3 tests have failures in areas that we consider significant for spec compliance.  I’ve listed proposed plans for all three below.  At least two recommend disabling the existing test.
      o   If the plans for these three test failures are adopted, then there would be no outstanding test to complete for submission.
·       readyState tests both passed on Firefox, which should mean that at-risk feature is considered as having a compliant implementation.
I’ve summarized tests with no passing implementation in this report, and for completeness (not compliance), have attached 3 FF specific test failures raised by Chris Pearce in an email this week.  We have a passing implementation on each of the 3 tests, but there are possible corrective actions in work for each.  They are listed at the bottom of the testing summary.

Reports:

·       http://w3c.github.io/test-results/encrypted-media/all.html


      o   Test files: 97; Total subtests: 306

·       http://w3c.github.io/test-results/encrypted-media/complete-fails.html

      o   Completely failed files: 14; Completely failed subtests: 14; Failure level: 14/306 (4.58%)

·       http://w3c.github.io/test-results/encrypted-media/less-than-2.html


      o   Test files without 2 passes: 28; Subtests without 2 passes: 40; Failure level: 40/306 (13.07%)

PRIORITY TESTS WITHOUT ONE PASSING IMPLEMENTATION (3 TESTS):
Persistent-license failures:
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total) OK
    drm, persistent-license, mp4, playback, check events      FAIL
        Status:  Will require pushing a new CDM.
        Plan:  Consider a bug with commitment to fix (https://crbug.com/660393)
Temporary session failures:
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)  OK
    drm, cannot load persistent license into temporary session FAIL
        Status:  Widevine license server converts license request to temporary.
        Plan:  Disable this test for V1.
/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)      OK
    drm, temporary, mp4, playback, check events      FAIL
        Status:  Will require pushing a new CDM.
        Plan:  Consider a bug with commitment to fix (https://crbug.com/660393)

TEST DETAILS:
Browser Versions:
CD56:  Chrome 56.0.2907.0 canary (64-bit)
CO56:  Chrome OS 56.0.2901.0 canary (64-bit)
ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
FF52:  Firefox 52.0a1 (2016-11-01) (32-bit)

Failure Summary:
================================================================================
Errors (considered bugs, not blocking)                                       0
Clear Key persistent-license (not planned, may make non-normative)           5
Clear Key persistent-usage-record (at risk, planned to be removed)           3
Clear Key transitions between clear and encrypted (bugs, not blocking)       0
Persistent-usage-record (at risk, planned to be removed)                     3
Clear Key transitions between clear and encrypted (bugs, not blocking)          0
================================================================================
Total Not Blocking Tests                                                    11
================================================================================
Persistent-license (fixes in work)                                           1
Temporary session edge cases (bugs, not block, fixes in work)                2
Temporary session readyState (resolved, no longer at risk)                   0
=================================================================================
Total Compliance Related Tests                                               3
TOTAL                                                                       14

Failed Tests by Group (14):
Error handling (considered bugs, not blocking) - 0
Clear key persistent-license tests (not planned to be implemented, may make informative note) - 5
/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-license, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-license, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
Clear key persistent-usage-record (not planned to be implemented) - 3
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-usage-record, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-usage-record, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
    org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window
Clear key transitions between clear and encrypted content (considered bugs, not blocking) - 0
Persistent-license failures - 1
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
    drm, persistent-license, mp4, playback, check events
Temporary session failures - 2
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
    drm, cannot load persistent license into temporary session
/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.33% of total)
    drm, temporary, mp4, playback, check events
Temporary session readyState failures (listed at risk, under discussion) – 0
================================================================================
Other FF Test Failures from Nov 1 email from Chris Pearce:
/encrypted-media/clearkey-mp4-unique-origin.html & /encrypted-media/drm-mp4-unique-origin.html
    Status:  Tests Chrome implementation, not the spec.
    Plan:  Disable the test for V1.
/encrypted-media/drm-events.html
    Status:  Test isn't completing in 10 s due to network latency.
    Plan:  Increase the latency.
/encrypted-media/drm-mp4-playback-temporary-expired.html
    Status:  Unknown issue.
    Plan:  Consider a bug with commitment to fix?  If so, can there be a public bug link?