Re: EME Test Status Update - 11/02/16

Hi Jerry,

I see that we have test results involving polyfills.

Do we explain those anywhere? Some of them are about 
persistent-usage-records so probably no longer relevant here. I'd like 
to make sure I represent those as accurately as possible.

Philippe


On 11/2/2016 7:15 PM, Jerry Smith (WPT) wrote:
> I’ve posted updated test status reports to EME, with two notable outcomes:
>
> ·       Only 3 tests have failures in areas that we consider significant
> for spec compliance.  I’ve listed proposed plans for all three below.
> At least two recommend disabling the existing test.
>
>       o   If the plans for these three test failures are adopted, then
> there would be no outstanding test to complete for submission.
>
> ·       */readyState/*tests both passed on Firefox, which should mean
> that at-risk feature is considered as having a compliant implementation.
>
> I’ve summarized tests with no passing implementation in this report, and
> for completeness (not compliance), have attached 3 FF specific test
> failures raised by Chris Pearce in an email this week.  We have a
> passing implementation on each of the 3 tests, but there are possible
> corrective actions in work for each.  They are listed at the bottom of
> the testing summary.
>
>
>
> *_Reports:_*
>
> ·       http://w3c.github.io/test-results/encrypted-media/all.html
>
>       o   *Test files*: 97; *Total subtests*: 306
>
> ·
> http://w3c.github.io/test-results/encrypted-media/complete-fails.html
>
>       o   *Completely failed files*: 14; *Completely failed subtests*:
> 14; *Failure level*: 14/306 (4.58%)
>
> ·       http://w3c.github.io/test-results/encrypted-media/less-than-2.html
>
>       o   *Test files without 2 passes*: 28; *Subtests without 2 passes:
> *40; *Failure level*: 40/306 (13.07%)
>
> * *
>
> *PRIORITY TESTS WITHOUT ONE PASSING IMPLEMENTATION (3 TESTS):     *
>
> *_Persistent-license failures:_*
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total) OK
>
>     drm, persistent-license, mp4, playback, check events      FAIL
>
>         Status:  Will require pushing a new CDM.
>
> */        Plan:  Consider a bug with commitment to fix
> (https://crbug.com/660393)      /*
>
> *_Temporary session failures:_*
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)  OK
>
>     drm, cannot load persistent license into temporary session FAIL
>
>         Status:  Widevine license server converts license request to
> temporary.
>
> */        Plan:  Disable this test for V1.  /*
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)      OK
>
>     drm, temporary, mp4, playback, check events      FAIL
>
>         Status:  Will require pushing a new CDM.
>
> */        Plan:  Consider a bug with commitment to fix
> (https://crbug.com/660393)      /*
>
>
>
> *TEST DETAILS:     *
>
> *_Browser Versions:_*
>
> CD56:  Chrome 56.0.2907.0 canary (64-bit)
>
> CO56:  Chrome OS 56.0.2901.0 canary (64-bit)
>
> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>
> FF52:  Firefox 52.0a1 (2016-11-01) (32-bit)
>
>
>
> *_Failure Summary:_*
>
> ================================================================================
>
> Errors (considered bugs, not
> blocking)                                       0
>
> Clear Key persistent-license (not planned, may make
> non-normative)           5
>
> Clear Key persistent-usage-record (at risk, planned to be
> removed)           3
>
> Clear Key transitions between clear and encrypted (bugs, not blocking)
>      0
>
> Persistent-usage-record (at risk, planned to be removed)
>       3
>
> Clear Key transitions between clear and encrypted (bugs, not
> blocking)          0
>
> ================================================================================
>
> */Total Not Blocking Tests
>                                                   11/*
>
> ================================================================================
>
> Persistent-license (fixes in work)
>                                         1
>
> Temporary session edge cases (bugs, not block, fixes in work)
>                2
>
> Temporary session readyState (resolved, no longer at risk)
>               0
>
> =================================================================================
>
> */Total Compliance Related Tests
>                    3/*
>
> *TOTAL
>     14*
>
> * *
>
> *_Failed Tests by Group (14):   _*
>
> *Error handling (considered bugs, not blocking) - 0*
>
> *Clear key persistent-license tests (not planned to be implemented, may
> make informative note) - 5*
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, destroy and
> acknowledge
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, retrieve,
> playback and destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, retrieve and
> playback
>
> *Clear key persistent-usage-record (not planned to be implemented) - 3*
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in
> new window
>
> *Clear key transitions between clear and encrypted content (considered
> bugs, not blocking) - 0*
>
> *Persistent-license failures - 1*
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.33% of total)
>
>     drm, persistent-license, mp4, playback, check events
>
> *Temporary session failures - 2*
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
> total)
>
>     drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)
>
>     drm, temporary, mp4, playback, check events
>
> *Temporary session readyState failures (listed at risk, under
> discussion) – 0*
>
> *================================================================================*
>
> *_Other FF Test Failures from Nov 1 email from Chris Pearce:_*
>
> /encrypted-media/clearkey-mp4-unique-origin.html &
> /encrypted-media/drm-mp4-unique-origin.html
>
>     Status:  Tests Chrome implementation, not the spec.
>
> /    *Plan:  Disable the test for V1.*/
>
> /encrypted-media/drm-events.html
>
>     Status:  Test isn't completing in 10 s due to network latency.
>
> */    Plan:  Increase the latency./*
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html
>
>     Status:  Unknown issue.
>
> */    Plan:  Consider a bug with commitment to fix?  If so, can there be
> a public bug link?/*
>
>
>

Received on Wednesday, 9 November 2016 20:32:20 UTC