- From: David Dorwin <ddorwin@google.com>
- Date: Tue, 4 Oct 2016 13:49:48 -0700
- To: Mark Watson <watsonm@netflix.com>
- Cc: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Matt Wolenetz <wolenetz@google.com>, Philippe Le Hégaret <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>
- Message-ID: <CAHD2rsggpLNN_9Q+Pfj12-W-booL0KMZUpmKVOVOPd6hCRObww@mail.gmail.com>
The tests need to be run using HTTPS. I can reproduce failures on HTTP. On Tue, Oct 4, 2016 at 1:00 PM, Mark Watson <watsonm@netflix.com> wrote: > There's still quite a lot here which "should" pass but which is failing. > We may have more regressions. > > ...Mark > > On Tue, Oct 4, 2016 at 11:40 AM, Jerry Smith (WPT) <jdsmith@microsoft.com> > wrote: > >> *General:* >> >> Most of the Google tests have been migrated, and all of those are now >> disabled in the repository. I ran the tests late yesterday, and am >> produced modified reports that show polyfill results IF the test fails for >> a given browser without the polyfill. If you see “Polyfill” in the report, >> it means just that. Polyfill results aren’t counted as a pass, and so show >> up in the failure reports as well. >> >> >> >> *Test reports:* >> >> - https://rawgit.com/w3c/test-results/gh-pages/encrypted-media >> /all.html >> >> - https://rawgit.com/w3c/test-results/gh-pages/encrypted-media >> /less-than-2.html >> >> - https://rawgit.com/w3c/test-results/gh-pages/encrypted-media >> /complete-fails.html >> >> >> >> *Summary: >> >> >> * >> >> Total TypeError failures: 14 >> tests Instead of InvalidAccessError >> >> Total Timeout issues: 17 >> tests Many on persistent session types or specific temporary >> session use cases >> >> Total Polyfill passes: 5 >> tests On persistent session types >> >> Total temporary playback with one pass: 11 tests >> Need to be fixed >> >> Total error case tests: 8 >> tests Are these tests necessary for CR? >> >> Total test scoring issues, not real problems: 14 tests >> One passing implementation because test is keySystem specific >> >> *Misc other failures: 11 >> tests >> >> * >> >> Total subtest failures: 80 >> tests >> >> >> >> *At Risk Based on Test:* >> >> 1. Temporary sessions >> >> 2. Persistent-license sessions >> >> 3. Persistent-usage-record sessions >> >> >> >> *Questions :* >> >> 1. When can temporary session type failures be fixed? >> >> 2. When will TypeError and promise rejection failures be fixed? >> >> 3. Are all error case tests required to prove the spec? >> >> >> >> *Details:* >> >> *1. **/encrypted-media/clearkey-generate-request-disallowed-input.html >> (7/7, 100.00%, 2.61% of total)* >> >> CH55 & FF52 Fail: TypeError expected, got InvalidAccessError (7 tests) >> >> *2. **/encrypted-media/clearkey-keystatuses.html (1/1, 100.00%, >> 0.37% of total)* >> >> CH55 Pass, FF 52 Fail: assert_equals: expected 0 but got 2 >> >> *3. **/encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-persistent-license-events.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-persistent-license.ht >> <http://playback-persistent-license.ht>ml (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-persistent-usage-record.html >> (1/1, 100.00%, 0.37% of total)* >> >> CH55 & FF%2 Pass with Polyfill – Known implementation issue, but include >> persistent sessions types that are not currently planned to be supported by >> all browsers (5 tests) >> >> *4. **/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-events.ht <http://drm-events.ht>ml (1/1, 100.00%, >> 0.37% of total)* >> >> */encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-expiration.html* >> >> CH55, FF52, ED14: Test Timeouts (7 tests) >> >> *5. **/encrypted-media/drm-generate-request-disallowed-input.html >> (7/7, 100.00%, 2.61% of total)* >> >> CH55, ED14, FF52: Most failures expect >> TypeError, got InValidAccessError (7 tests) >> >> *6. **/encrypted-media/drm-keystatuses-multiple-sessions.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, >> 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-persistent-usage-record-events.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html >> (1/1, 100.00%, 0.37% of total)* >> >> CH55, ED14, FF52: Mostly Timeout (7 tests) >> >> *7. **/encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%, >> 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of >> total)* >> >> CH55 & ED14 Timeout, FF52 Pass: All playback tests with MP4 content (11 >> tests) >> >> *8. **/encrypted-media/drm-temporary-license-type.html (1/1, >> 100.00%, 0.37% of total)* >> >> CH55 & ED14 Timeout, FF52 Fail: Error case, shouldn’t play with >> persistent license, but does (1 test) >> >> *9. **/encrypted-media/idlharness.ht <http://idlharness.ht>ml >> (10/58, 17.24%, 3.73% of total)* >> >> CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52 >> Pass (9 tests) >> >> CH55, ED14, FF52 Fail: MediaKeyMessageEvent.length expected 2, but got 1 >> (1 test) >> >> *10. **/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.ht >> <http://requestmediakeysystemaccess.ht>ml (4/58, 6.90%, 1.49% of total)* >> >> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax >> checks/error cases (4 tests) >> >> *11. **/encrypted-media/drm-mp4-playback-persistent-usage-record.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html >> (1/1, 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, >> 100.00%, 0.37% of total)* >> >> CH55, ED14, FF52: Timeout (3 tests) >> >> *12. **/encrypted-media/drm-mp4-playback-temporary-events.html (1/1, >> 100.00%, 0.37% of total)* >> >> */encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html >> (1/1, 100.00%, 0.37% of total)* >> >> CH55 & ED14: Timeout (2 tests) >> >> FF52: Fail one test for sequence length of 9, 10 expected >> >> FF52: Fail one test for Promise still outstanding at MediaKeys shutdown >> >> *13. **/encrypted-media/drm-mp4-requestmediakeysystemaccess.html >> (17/72, 23.61%, 6.34% of total* >> >> CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax >> checks/error cases (3 tests) >> >> ED14 only Pass: Test scoring issue, not valid single implementation >> feature (14 tests) >> >> Jerry >> > >
Received on Tuesday, 4 October 2016 20:50:39 UTC