W3C home > Mailing lists > Public > public-html-media@w3.org > October 2016

Re: Updated EME Test Results

From: David Dorwin <ddorwin@google.com>
Date: Tue, 4 Oct 2016 13:49:48 -0700
Message-ID: <CAHD2rsggpLNN_9Q+Pfj12-W-booL0KMZUpmKVOVOPd6hCRObww@mail.gmail.com>
To: Mark Watson <watsonm@netflix.com>
Cc: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Matt Wolenetz <wolenetz@google.com>, Philippe Le Hégaret <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>
The tests need to be run using HTTPS. I can reproduce failures on HTTP.

On Tue, Oct 4, 2016 at 1:00 PM, Mark Watson <watsonm@netflix.com> wrote:

> There's still quite a lot here which "should" pass but which is failing.
> We may have more regressions.
>
> ...Mark
>
> On Tue, Oct 4, 2016 at 11:40 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
>> *General:*
>>
>> Most of the Google tests have been migrated, and all of those are now
>> disabled in the repository.  I ran the tests late yesterday, and am
>> produced modified reports that show polyfill results IF the test fails for
>> a given browser without the polyfill.  If you see “Polyfill” in the report,
>> it means just that.  Polyfill results aren’t counted as a pass, and so show
>> up in the failure reports as well.
>>
>>
>>
>> *Test reports:*
>>
>> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media
>> /all.html
>>
>> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media
>> /less-than-2.html
>>
>> -          https://rawgit.com/w3c/test-results/gh-pages/encrypted-media
>> /complete-fails.html
>>
>>
>>
>> *Summary:
>>
>>
>>                                                        *
>>
>> Total TypeError failures:                                         14
>> tests               Instead of InvalidAccessError
>>
>> Total Timeout issues:                                              17
>> tests               Many on persistent session types or specific temporary
>> session use cases
>>
>> Total Polyfill passes:                                                5
>> tests                 On persistent session types
>>
>> Total temporary playback with one pass:             11 tests
>>               Need to be fixed
>>
>> Total error case tests:                                             8
>> tests                 Are these tests necessary for CR?
>>
>> Total test scoring issues, not real problems:         14 tests
>>               One passing implementation because test is keySystem specific
>>
>> *Misc other failures:                                                 11
>> tests
>>
>>                                                        *
>>
>> Total subtest failures:                                              80
>> tests
>>
>>
>>
>> *At Risk Based on Test:*
>>
>> 1.       Temporary sessions
>>
>> 2.       Persistent-license sessions
>>
>> 3.       Persistent-usage-record sessions
>>
>>
>>
>> *Questions :*
>>
>> 1.       When can temporary session type failures be fixed?
>>
>> 2.       When will TypeError and promise rejection failures be fixed?
>>
>> 3.       Are all error case tests required to prove the spec?
>>
>>
>>
>> *Details:*
>>
>> *1.       **/encrypted-media/clearkey-generate-request-disallowed-input.html
>> (7/7, 100.00%, 2.61% of total)*
>>
>> CH55 & FF52 Fail:  TypeError expected, got InvalidAccessError (7 tests)
>>
>> *2.       **/encrypted-media/clearkey-keystatuses.html (1/1, 100.00%,
>> 0.37% of total)*
>>
>> CH55 Pass, FF 52 Fail:  assert_equals: expected 0 but got 2
>>
>> *3.       **/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-persistent-license.ht
>> <http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> CH55 & FF%2 Pass with Polyfill – Known implementation issue, but include
>> persistent sessions types that are not currently planned to be supported by
>> all browsers (5 tests)
>>
>> *4.       **/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-events.ht <http://drm-events.ht>ml (1/1, 100.00%,
>> 0.37% of total)*
>>
>> */encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-expiration.html*
>>
>> CH55, FF52, ED14:  Test Timeouts (7 tests)
>>
>> *5.       **/encrypted-media/drm-generate-request-disallowed-input.html
>> (7/7, 100.00%, 2.61% of total)*
>>
>>                            CH55, ED14, FF52:  Most failures expect
>> TypeError, got InValidAccessError (7 tests)
>>
>> *6.       **/encrypted-media/drm-keystatuses-multiple-sessions.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%,
>> 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> CH55, ED14, FF52: Mostly Timeout (7 tests)
>>
>> *7.       **/encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%,
>> 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of
>> total)*
>>
>> CH55 & ED14 Timeout, FF52 Pass:  All playback tests with MP4 content (11
>> tests)
>>
>> *8.       **/encrypted-media/drm-temporary-license-type.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> CH55 & ED14 Timeout, FF52 Fail:  Error case, shouldn’t play with
>> persistent license, but does (1 test)
>>
>> *9.       **/encrypted-media/idlharness.ht <http://idlharness.ht>ml
>> (10/58, 17.24%, 3.73% of total)*
>>
>> CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52
>> Pass (9 tests)
>>
>> CH55, ED14, FF52 Fail:  MediaKeyMessageEvent.length expected 2, but got 1
>> (1 test)
>>
>> *10.   **/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.ht
>> <http://requestmediakeysystemaccess.ht>ml (4/58, 6.90%, 1.49% of total)*
>>
>> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
>> checks/error cases (4 tests)
>>
>> *11.   **/encrypted-media/drm-mp4-playback-persistent-usage-record.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> CH55, ED14, FF52: Timeout (3 tests)
>>
>> *12.   **/encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
>> 100.00%, 0.37% of total)*
>>
>> */encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.37% of total)*
>>
>> CH55 & ED14: Timeout (2 tests)
>>
>> FF52: Fail one test for sequence length of 9, 10 expected
>>
>> FF52: Fail one test for Promise still outstanding at MediaKeys shutdown
>>
>> *13.   **/encrypted-media/drm-mp4-requestmediakeysystemaccess.html
>> (17/72, 23.61%, 6.34% of total*
>>
>> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax
>> checks/error cases (3 tests)
>>
>> ED14 only Pass: Test scoring issue, not valid single implementation
>> feature (14 tests)
>>
>> Jerry
>>
>
>
Received on Tuesday, 4 October 2016 20:50:39 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 4 October 2016 20:50:40 UTC