W3C home > Mailing lists > Public > public-html-media@w3.org > October 2016

Re: Updated EME Test Results

From: Mark Watson <watsonm@netflix.com>
Date: Tue, 4 Oct 2016 13:00:20 -0700
Message-ID: <CAEnTvdAoS7mQU+_+qcKc7AAagf1vAW-oiuD3njjRd4vJKpiQ6Q@mail.gmail.com>
To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>, David Dorwin <ddorwin@google.com>, Matt Wolenetz <wolenetz@google.com>, Philippe Le Hégaret <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>
There's still quite a lot here which "should" pass but which is failing. We
may have more regressions.

...Mark

On Tue, Oct 4, 2016 at 11:40 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> *General:*
>
> Most of the Google tests have been migrated, and all of those are now
> disabled in the repository.  I ran the tests late yesterday, and am
> produced modified reports that show polyfill results IF the test fails for
> a given browser without the polyfill.  If you see “Polyfill” in the report,
> it means just that.  Polyfill results aren’t counted as a pass, and so show
> up in the failure reports as well.
>
>
>
> *Test reports:*
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/all.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/less-than-2.html
>
> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-
> media/complete-fails.html
>
>
>
> *Summary:
>
>
>                                                        *
>
> Total TypeError failures:                                         14
> tests               Instead of InvalidAccessError
>
> Total Timeout issues:                                              17
> tests               Many on persistent session types or specific temporary
> session use cases
>
> Total Polyfill passes:                                                5
> tests                 On persistent session types
>
> Total temporary playback with one pass:             11 tests
> Need to be fixed
>
> Total error case tests:                                             8
> tests                 Are these tests necessary for CR?
>
> Total test scoring issues, not real problems:         14 tests
>               One passing implementation because test is keySystem specific
>
> *Misc other failures:                                                 11
> tests
>
>                                                        *
>
> Total subtest failures:                                              80
> tests
>
>
>
> *At Risk Based on Test:*
>
> 1.       Temporary sessions
>
> 2.       Persistent-license sessions
>
> 3.       Persistent-usage-record sessions
>
>
>
> *Questions :*
>
> 1.       When can temporary session type failures be fixed?
>
> 2.       When will TypeError and promise rejection failures be fixed?
>
> 3.       Are all error case tests required to prove the spec?
>
>
>
> *Details:*
>
> *1.       **/encrypted-media/clearkey-generate-request-disallowed-input.html
> (7/7, 100.00%, 2.61% of total)*
>
> CH55 & FF52 Fail:  TypeError expected, got InvalidAccessError (7 tests)
>
> *2.       **/encrypted-media/clearkey-keystatuses.html (1/1, 100.00%,
> 0.37% of total)*
>
> CH55 Pass, FF 52 Fail:  assert_equals: expected 0 but got 2
>
> *3.       **/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-license-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55 & FF%2 Pass with Polyfill – Known implementation issue, but include
> persistent sessions types that are not currently planned to be supported by
> all browsers (5 tests)
>
> *4.       **/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-events.html (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-expiration.html*
>
> CH55, FF52, ED14:  Test Timeouts (7 tests)
>
> *5.       **/encrypted-media/drm-generate-request-disallowed-input.html
> (7/7, 100.00%, 2.61% of total)*
>
>                            CH55, ED14, FF52:  Most failures expect
> TypeError, got InValidAccessError (7 tests)
>
> *6.       **/encrypted-media/drm-keystatuses-multiple-sessions.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
> 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> CH55, ED14, FF52: Mostly Timeout (7 tests)
>
> *7.       **/encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%,
> 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of
> total)*
>
> CH55 & ED14 Timeout, FF52 Pass:  All playback tests with MP4 content (11
> tests)
>
> *8.       **/encrypted-media/drm-temporary-license-type.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55 & ED14 Timeout, FF52 Fail:  Error case, shouldn’t play with
> persistent license, but does (1 test)
>
> *9.       **/encrypted-media/idlharness.html (10/58, 17.24%, 3.73% of
> total)*
>
> CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52 Pass
> (9 tests)
>
> CH55, ED14, FF52 Fail:  MediaKeyMessageEvent.length expected 2, but got 1
> (1 test)
>
> *10.   **/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html
> (4/58, 6.90%, 1.49% of total)*
>
> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
> checks/error cases (4 tests)
>
> *11.   **/encrypted-media/drm-mp4-playback-persistent-usage-record.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
> (1/1, 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
> 100.00%, 0.37% of total)*
>
> CH55, ED14, FF52: Timeout (3 tests)
>
> *12.   **/encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
> 100.00%, 0.37% of total)*
>
> */encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
> (1/1, 100.00%, 0.37% of total)*
>
> CH55 & ED14: Timeout (2 tests)
>
> FF52: Fail one test for sequence length of 9, 10 expected
>
> FF52: Fail one test for Promise still outstanding at MediaKeys shutdown
>
> *13.   **/encrypted-media/drm-mp4-requestmediakeysystemaccess.html
> (17/72, 23.61%, 6.34% of total*
>
> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
> checks/error cases (3 tests)
>
> ED14 only Pass: Test scoring issue, not valid single implementation
> feature (14 tests)
>
> Jerry
>
Received on Tuesday, 4 October 2016 20:00:49 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 15:49:17 UTC