W3C home > Mailing lists > Public > public-html-media@w3.org > October 2016

RE: Updated EME Test Results

From: Jerry Smith (WPT) <jdsmith@microsoft.com>
Date: Wed, 5 Oct 2016 00:04:55 +0000
To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, David Dorwin <ddorwin@google.com>, Mark Watson <watsonm@netflix.com>
CC: Paul Cotton <Paul.Cotton@microsoft.com>, Matt Wolenetz <wolenetz@google.com>, Philippe Le Hégaret <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>
Message-ID: <SN2PR03MB04740E9943937769E788EE6A4C40@SN2PR03MB047.namprd03.prod.outlook.com>
I can only run https on the public test harness link (https://w3c-test.org/tools/runner/index.html) which means without polyfills unless we merge them there.


-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html

-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/less-than-2.html

-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/complete-fails.html

I don't get dramatically more passes using https, but there are also new tests added, so the numbers flexed from that as well.


-          All:

o   Was:     Completely failed files: undefined; Completely failed subtests: 40; Failure level: 40/268 (14.93%)

o   Now:     Completely failed files: undefined; Completely failed subtests: 43; Failure level: 43/277 (15.52%)

-          Less than 2:

o   Was:     Test files without 2 passes: 36; Subtests without 2 passes: 76; Failure level: 76/277 (27.44%)

o   Now:     Test files without 2 passes: 40; Subtests without 2 passes: 80; Failure level: 80/268 (29.85%)

-          Complete fail:

o   Was:     Test files: 63; Total subtests: 268 (Polyfills double the test count, so this is actually 138 subtests total)

o   Now:     Test files: 72; Total subtests: 277

"Now" results are from the new pass using https.  I do see these tests now passing on CH55 (at least) that didn't previously:

/encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-multikey.html>

drm, temporary, mp4, playback, multikey audio/video

/encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-multisession.html>

drm, temporary, mp4, playback with multiple sessions, multikey video

/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html>

drm, temporary, mp4, playback, setMediaKeys after setting video.src

/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html>

drm, temporary, mp4, playback, setMediaKeys after updating session

/encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-two-videos.html>

drm, temporary, mp4, playback two videos

/encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary-waitingforkey.html>

drm, successful playback, temporary, mp4, waitingforkey event, 1 key

/encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of total)<http://www.w3c-test.org/encrypted-media/drm-mp4-playback-temporary.html>

drm, temporary, mp4, playback, single key


It seems from here out we should run tests on the public harness.  Do we want to merge in the polyfilled tests (and update them each merge)?  Mark:  What's your take?  My read is that the Polyfills are mostly affecting persistent sessions that will likely be put at risk.

Jerry

From: Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
Sent: Tuesday, October 4, 2016 2:29 PM
To: David Dorwin <ddorwin@google.com>; Mark Watson <watsonm@netflix.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>; Matt Wolenetz <wolenetz@google.com>; Philippe Le Hégaret <plh@w3.org>; public-html-media@w3.org
Subject: RE: Updated EME Test Results

Thanks.  I will rerun and post uodated reports a bit later today.

Sent from my Windows 10 phone

From: David Dorwin<mailto:ddorwin@google.com>
Sent: Tuesday, October 4, 2016 1:50 PM
To: Mark Watson<mailto:watsonm@netflix.com>
Cc: Jerry Smith (WPT)<mailto:jdsmith@microsoft.com>; Paul Cotton<mailto:Paul.Cotton@microsoft.com>; Matt Wolenetz<mailto:wolenetz@google.com>; Philippe Le Hégaret<mailto:plh@w3.org>; public-html-media@w3.org<mailto:public-html-media@w3.org>
Subject: Re: Updated EME Test Results

The tests need to be run using HTTPS. I can reproduce failures on HTTP.

On Tue, Oct 4, 2016 at 1:00 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
There's still quite a lot here which "should" pass but which is failing. We may have more regressions.

...Mark

On Tue, Oct 4, 2016 at 11:40 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
General:
Most of the Google tests have been migrated, and all of those are now disabled in the repository.  I ran the tests late yesterday, and am produced modified reports that show polyfill results IF the test fails for a given browser without the polyfill.  If you see "Polyfill" in the report, it means just that.  Polyfill results aren't counted as a pass, and so show up in the failure reports as well.

Test reports:

-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html

-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/less-than-2.html

-          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/complete-fails.html

Summary:
Total TypeError failures:                                         14 tests               Instead of InvalidAccessError
Total Timeout issues:                                              17 tests               Many on persistent session types or specific temporary session use cases
Total Polyfill passes:                                                5 tests                 On persistent session types
Total temporary playback with one pass:             11 tests               Need to be fixed
Total error case tests:                                             8 tests                 Are these tests necessary for CR?
Total test scoring issues, not real problems:         14 tests               One passing implementation because test is keySystem specific
Misc other failures:                                                 11 tests
Total subtest failures:                                              80 tests

At Risk Based on Test:

1.       Temporary sessions

2.       Persistent-license sessions

3.       Persistent-usage-record sessions

Questions :

1.       When can temporary session type failures be fixed?

2.       When will TypeError and promise rejection failures be fixed?

3.       Are all error case tests required to prove the spec?

Details:

1.       /encrypted-media/clearkey-generate-request-disallowed-input.html (7/7, 100.00%, 2.61% of total)

CH55 & FF52 Fail:  TypeError expected, got InvalidAccessError (7 tests)

2.       /encrypted-media/clearkey-keystatuses.html (1/1, 100.00%, 0.37% of total)

CH55 Pass, FF 52 Fail:  assert_equals: expected 0 but got 2

3.       /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)

/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-license.ht<http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
CH55 & FF%2 Pass with Polyfill - Known implementation issue, but include persistent sessions types that are not currently planned to be supported by all browsers (5 tests)

4.       /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-events.ht<http://drm-events.ht>ml (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-expiration.html
CH55, FF52, ED14:  Test Timeouts (7 tests)

5.       /encrypted-media/drm-generate-request-disallowed-input.html (7/7, 100.00%, 2.61% of total)
                           CH55, ED14, FF52:  Most failures expect TypeError, got InValidAccessError (7 tests)

6.       /encrypted-media/drm-keystatuses-multiple-sessions.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
CH55, ED14, FF52: Mostly Timeout (7 tests)

7.       /encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14 Timeout, FF52 Pass:  All playback tests with MP4 content (11 tests)

8.       /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14 Timeout, FF52 Fail:  Error case, shouldn't play with persistent license, but does (1 test)

9.       /encrypted-media/idlharness.ht<http://idlharness.ht>ml (10/58, 17.24%, 3.73% of total)
CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52 Pass (9 tests)
CH55, ED14, FF52 Fail:  MediaKeyMessageEvent.length expected 2, but got 1 (1 test)

10.   /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.ht<http://requestmediakeysystemaccess.ht>ml (4/58, 6.90%, 1.49% of total)
CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax checks/error cases (4 tests)

11.   /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.37% of total)
CH55, ED14, FF52: Timeout (3 tests)

12.   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14: Timeout (2 tests)
FF52: Fail one test for sequence length of 9, 10 expected
FF52: Fail one test for Promise still outstanding at MediaKeys shutdown

13.   /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (17/72, 23.61%, 6.34% of total
CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax checks/error cases (3 tests)
ED14 only Pass: Test scoring issue, not valid single implementation feature (14 tests)
Jerry
Received on Wednesday, 5 October 2016 00:05:35 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 5 October 2016 00:05:37 UTC