Re: Updated EME Test Results

On Tue, Oct 4, 2016 at 5:04 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> I can only run https on the public test harness link (
> https://w3c-test.org/tools/runner/index.html) which means without
> polyfills unless we merge them there.
>

The test server that you can run locally supports https. You just need to
add a security exception in the browser (when you visit the page).

...Mark



>
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/all.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/less-than-2.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/complete-fails.html
>
>
>
> I don’t get dramatically more passes using https, but there are also new
> tests added, so the numbers flexed from that as well.
>
>
>
> -          All:
>
> o   Was:     *Completely failed files*: undefined; *Completely failed
> subtests*: 40; *Failure level*: 40/268 (14.93%)
>
> o   Now:     *Completely failed files*: undefined; *Completely failed
> subtests*: 43; *Failure level*: 43/277 (15.52%)
>
> -          Less than 2:
>
> o   Was:     *Test files without 2 passes*: 36; *Subtests without 2
> passes: *76; *Failure level*: 76/277 (27.44%)
>
> o   Now:     *Test files without 2 passes*: 40; *Subtests without 2
> passes: *80; *Failure level*: 80/268 (29.85%)
>
> -          Complete fail:
>
> o   Was:     *Test files*: 63; *Total subtests*: 268 (Polyfills double
> the test count, so this is actually 138 subtests total)
>
> o   Now:     *Test files*: 72; *Total subtests*: 277
>
>
>
> “Now” results are from the new pass using https.  I do see these tests now
> passing on CH55 (at least) that didn’t previously:
>
>
>
> */encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%,
> 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-multikey.html>*
>
> drm, temporary, mp4, playback, multikey audio/video
>
> */encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1,
> 100.00%, 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-multisession.html>*
>
> drm, temporary, mp4, playback with multiple sessions, multikey video
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html
> (1/1, 100.00%, 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html>*
>
> drm, temporary, mp4, playback, setMediaKeys after setting video.src
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html
> (1/1, 100.00%, 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html>*
>
> drm, temporary, mp4, playback, setMediaKeys after updating session
>
> */encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1,
> 100.00%, 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-two-videos.html>*
>
> drm, temporary, mp4, playback two videos
>
> */encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1,
> 100.00%, 0.37% of total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-waitingforkey.html>*
>
> drm, successful playback, temporary, mp4, waitingforkey event, 1 key
>
> */encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of
> total)
> <http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary.html>*
>
> drm, temporary, mp4, playback, single key
>
>
>
> It seems from here out we should run tests on the public harness.  Do we
> want to merge in the polyfilled tests (and update them each merge)?  Mark:
> What’s your take?  My read is that the Polyfills are mostly affecting
> persistent sessions that will likely be put at risk.
>
>
>
> Jerry
>
>
>
> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
> *Sent:* Tuesday, October 4, 2016 2:29 PM
> *To:* David Dorwin <ddorwin@google.com>; Mark Watson <watsonm@netflix.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Matt Wolenetz <
> wolenetz@google.com>; Philippe Le Hégaret <plh@w3.org>;
> public-html-media@w3.org
> *Subject:* RE: Updated EME Test Results
>
>
>
> Thanks.  I will rerun and post uodated reports a bit later today.
>
>
>
> Sent from my Windows 10 phone
>
>
>
> *From: *David Dorwin <ddorwin@google.com>
> *Sent: *Tuesday, October 4, 2016 1:50 PM
> *To: *Mark Watson <watsonm@netflix.com>
> *Cc: *Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton
> <Paul.Cotton@microsoft.com>; Matt Wolenetz <wolenetz@google.com>; Philippe
> Le Hégaret <plh@w3.org>; public-html-media@w3.org
> *Subject: *Re: Updated EME Test Results
>
>
>
> The tests need to be run using HTTPS. I can reproduce failures on HTTP.
>
>
>
> On Tue, Oct 4, 2016 at 1:00 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> There's still quite a lot here which "should" pass but which is failing.
> We may have more regressions.
>
>
>
> ...Mark
>
>
>
> On Tue, Oct 4, 2016 at 11:40 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> *General:*
>
> Most of the Google tests have been migrated, and all of those are now
> disabled in the repository.  I ran the tests late yesterday, and am
> produced modified reports that show polyfill results IF the test fails for
> a given browser without the polyfill.  If you see “Polyfill” in the report,
> it means just that.  Polyfill results aren’t counted as a pass, and so show
> up in the failure reports as well.
>
>
>
> *Test reports:*
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/all.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/less-than-2.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/complete-fails.html
>
>
>
> *Summary:
>
>
>                                                        *
>
> Total TypeError failures:                                         14
> tests               Instead of InvalidAccessError
>
> Total Timeout issues:                                              17
> tests               Many on persistent session types or specific temporary
> session use cases
>
> Total Polyfill passes:                                                5
> tests                 On persistent session types
>
> Total temporary playback with one pass:             11 tests
> Need to be fixed
>
> Total error case tests:                                             8
> tests                 Are these tests necessary for CR?
>
> Total test scoring issues, not real problems:         14 tests
>               One passing implementation because test is keySystem specific
>
> *Misc other failures:                                                 11
> tests
>
>                                                        *
>
> Total subtest failures:                                              80
> tests
>
>
>
> *At Risk Based on Test:*
>
> 1.       Temporary sessions
>
> 2.       Persistent-license sessions
>
> 3.       Persistent-usage-record sessions
>
>
>
> *Questions :*
>
> 1.       When can temporary session type failures be fixed?
>
> 2.       When will TypeError and promise rejection failures be fixed?
>
> 3.       Are all error case tests required to prove the spec?
>
>
>
> *Details:*
>
> *1.*       */encrypted-media/clearkey-generate-request-disallowed-input.html
> (7/7, 100.00%, 2.61% of total)*
>
> CH55 & FF52 Fail:  TypeError expected, got InvalidAccessError (7 tests)
>
> *2.*       */encrypted-media/clearkey-keystatuses.html (1/1, 100.00%,
> 0.37% of total)*
>
> CH55 Pass, FF 52 Fail:  assert_equals: expected 0 but got 2
>
> *3.*       */encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-license.ht
> <http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55 & FF%2 Pass with Polyfill – Known implementation issue, but include
> persistent sessions types that are not currently planned to be supported by
> all browsers (5 tests)
>
> *4.*       */encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-events.ht <http://drm-events.ht>ml (1/1, 100.00%,
> 0.37% of total)*
>
> */encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-expiration.html*
>
> CH55, FF52, ED14:  Test Timeouts (7 tests)
>
> *5.*       */encrypted-media/drm-generate-request-disallowed-input.html
> (7/7, 100.00%, 2.61% of total)*
>
>                            CH55, ED14, FF52:  Most failures expect
> TypeError, got InValidAccessError (7 tests)
>
> *6.*       */encrypted-media/drm-keystatuses-multiple-sessions.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> CH55, ED14, FF52: Mostly Timeout (7 tests)
>
> *7.*       */encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%,
> 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of
> total)*
>
> CH55 & ED14 Timeout, FF52 Pass:  All playback tests with MP4 content (11
> tests)
>
> *8.*       */encrypted-media/drm-temporary-license-type.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55 & ED14 Timeout, FF52 Fail:  Error case, shouldn’t play with
> persistent license, but does (1 test)
>
> *9.*       */encrypted-media/idlharness.ht <http://idlharness.ht>ml
> (10/58, 17.24%, 3.73% of total)*
>
> CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52 Pass
> (9 tests)
>
> CH55, ED14, FF52 Fail:  MediaKeyMessageEvent.length expected 2, but got 1
> (1 test)
>
> *10.*   */encrypted-media/clearkey-mp4-requestmediakeysystemaccess.ht
> <http://requestmediakeysystemaccess.ht>ml (4/58, 6.90%, 1.49% of total)*
>
> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
> checks/error cases (4 tests)
>
> *11.*   */encrypted-media/drm-mp4-playback-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55, ED14, FF52: Timeout (3 tests)
>
> *12.*   */encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.37% of total)*
>
> CH55 & ED14: Timeout (2 tests)
>
> FF52: Fail one test for sequence length of 9, 10 expected
>
> FF52: Fail one test for Promise still outstanding at MediaKeys shutdown
>
> *13.*   */encrypted-media/drm-mp4-requestmediakeysystemaccess.html
> (17/72, 23.61%, 6.34% of total*
>
> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
> checks/error cases (3 tests)
>
> ED14 only Pass: Test scoring issue, not valid single implementation
> feature (14 tests)
>
> Jerry
>
>
>
>
>