RE: EME Test Status Update - 11/02/16

Also, the only tests affected by polyfills are persistent-license ones with clearkey.  None of the browsers support this scenario now, but it is conceptually possible via site javascript.  The polyfills essentially demonstrate that possibility.

Jerry

-----Original Message-----
From: Jerry Smith (WPT) 
Sent: Wednesday, November 9, 2016 1:52 PM
To: 'Philippe Le Hégaret' <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>; public-html-media@w3.org
Subject: RE: EME Test Status Update - 11/02/16

I added this text at the top of each report:

"A "POLYFILL" result indicates a test that only passed when polyfills were used to compensate for browser implementation issues. Polyfills were used only where implementation fixes weren't possible now, but believed to be straightforward."

So to be clear:

- Polyfill tests are duplicates to the main tests with polyfills added in - both the regular and polyfill versions of each test are run.
- If the test passes on its own, polyfill test results are ignored.
- If the test fails on its own, polyfill results are checked - if they pass, then the report gives a green result with "POLYFILL" text.

I can easily add to that text & repost the reports, if you think it needs to say more.

Jerry

-----Original Message-----
From: Philippe Le Hégaret [mailto:plh@w3.org]
Sent: Wednesday, November 9, 2016 12:32 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <Paul.Cotton@microsoft.com>; public-html-media@w3.org
Subject: Re: EME Test Status Update - 11/02/16

Hi Jerry,

I see that we have test results involving polyfills.

Do we explain those anywhere? Some of them are about persistent-usage-records so probably no longer relevant here. I'd like to make sure I represent those as accurately as possible.

Philippe


On 11/2/2016 7:15 PM, Jerry Smith (WPT) wrote:
> I’ve posted updated test status reports to EME, with two notable outcomes:
>
> ·       Only 3 tests have failures in areas that we consider significant
> for spec compliance.  I’ve listed proposed plans for all three below.
> At least two recommend disabling the existing test.
>
>       o   If the plans for these three test failures are adopted, then
> there would be no outstanding test to complete for submission.
>
> ·       */readyState/*tests both passed on Firefox, which should mean
> that at-risk feature is considered as having a compliant implementation.
>
> I’ve summarized tests with no passing implementation in this report, 
> and for completeness (not compliance), have attached 3 FF specific 
> test failures raised by Chris Pearce in an email this week.  We have a 
> passing implementation on each of the 3 tests, but there are possible 
> corrective actions in work for each.  They are listed at the bottom of 
> the testing summary.
>
>
>
> *_Reports:_*
>
> ·       http://w3c.github.io/test-results/encrypted-media/all.html

>
>       o   *Test files*: 97; *Total subtests*: 306
>
> ·
> http://w3c.github.io/test-results/encrypted-media/complete-fails.html

>
>       o   *Completely failed files*: 14; *Completely failed subtests*:
> 14; *Failure level*: 14/306 (4.58%)
>
> ·       http://w3c.github.io/test-results/encrypted-media/less-than-2.html

>
>       o   *Test files without 2 passes*: 28; *Subtests without 2 passes:
> *40; *Failure level*: 40/306 (13.07%)
>
> * *
>
> *PRIORITY TESTS WITHOUT ONE PASSING IMPLEMENTATION (3 TESTS):     *
>
> *_Persistent-license failures:_*
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 
> 100.00%, 0.33% of total) OK
>
>     drm, persistent-license, mp4, playback, check events      FAIL
>
>         Status:  Will require pushing a new CDM.
>
> */        Plan:  Consider a bug with commitment to fix
> (https://crbug.com/660393)      /*
>
> *_Temporary session failures:_*
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% 
> of
> total)  OK
>
>     drm, cannot load persistent license into temporary session FAIL
>
>         Status:  Widevine license server converts license request to 
> temporary.
>
> */        Plan:  Disable this test for V1.  /*
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
> 0.33% of total)      OK
>
>     drm, temporary, mp4, playback, check events      FAIL
>
>         Status:  Will require pushing a new CDM.
>
> */        Plan:  Consider a bug with commitment to fix
> (https://crbug.com/660393)      /*
>
>
>
> *TEST DETAILS:     *
>
> *_Browser Versions:_*
>
> CD56:  Chrome 56.0.2907.0 canary (64-bit)
>
> CO56:  Chrome OS 56.0.2901.0 canary (64-bit)
>
> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>
> FF52:  Firefox 52.0a1 (2016-11-01) (32-bit)
>
>
>
> *_Failure Summary:_*
>
> ======================================================================
> ==========
>
> Errors (considered bugs, not
> blocking)                                       0
>
> Clear Key persistent-license (not planned, may make
> non-normative)           5
>
> Clear Key persistent-usage-record (at risk, planned to be
> removed)           3
>
> Clear Key transitions between clear and encrypted (bugs, not blocking)
>      0
>
> Persistent-usage-record (at risk, planned to be removed)
>       3
>
> Clear Key transitions between clear and encrypted (bugs, not
> blocking)          0
>
> ======================================================================
> ==========
>
> */Total Not Blocking Tests
>                                                   11/*
>
> ======================================================================
> ==========
>
> Persistent-license (fixes in work)
>                                         1
>
> Temporary session edge cases (bugs, not block, fixes in work)
>                2
>
> Temporary session readyState (resolved, no longer at risk)
>               0
>
> ======================================================================
> ===========
>
> */Total Compliance Related Tests
>                    3/*
>
> *TOTAL
>     14*
>
> * *
>
> *_Failed Tests by Group (14):   _*
>
> *Error handling (considered bugs, not blocking) - 0*
>
> *Clear key persistent-license tests (not planned to be implemented, 
> may make informative note) - 5*
>
> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, destroy and 
> acknowledge
>
> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, check events
>
> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 
> 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-lic
> ense.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, retrieve, 
> playback and destroy
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.htm
> l
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-license, mp4, playback, retrieve and 
> playback
>
> *Clear key persistent-usage-record (not planned to be implemented) -
> 3*
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.
> html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4, playback, check 
> events
>
> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4playback
>
> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-recor
> d.html
> (1/1, 100.00%, 0.33% of total)
>
>     org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve 
> in new window
>
> *Clear key transitions between clear and encrypted content (considered 
> bugs, not blocking) - 0*
>
> *Persistent-license failures - 1*
>
> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 
> 100.00%, 0.33% of total)
>
>     drm, persistent-license, mp4, playback, check events
>
> *Temporary session failures - 2*
>
> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% 
> of
> total)
>
>     drm, cannot load persistent license into temporary session
>
> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 
> 0.33% of total)
>
>     drm, temporary, mp4, playback, check events
>
> *Temporary session readyState failures (listed at risk, under
> discussion) – 0*
>
> *=====================================================================
> ===========*
>
> *_Other FF Test Failures from Nov 1 email from Chris Pearce:_*
>
> /encrypted-media/clearkey-mp4-unique-origin.html & 
> /encrypted-media/drm-mp4-unique-origin.html
>
>     Status:  Tests Chrome implementation, not the spec.
>
> /    *Plan:  Disable the test for V1.*/
>
> /encrypted-media/drm-events.html
>
>     Status:  Test isn't completing in 10 s due to network latency.
>
> */    Plan:  Increase the latency./*
>
> /encrypted-media/drm-mp4-playback-temporary-expired.html
>
>     Status:  Unknown issue.
>
> */    Plan:  Consider a bug with commitment to fix?  If so, can there be
> a public bug link?/*
>
>
>

Received on Wednesday, 9 November 2016 21:56:54 UTC