- From: Jerry Smith (WPT) <jdsmith@microsoft.com>
- Date: Tue, 4 Oct 2016 18:40:18 +0000
- To: Paul Cotton <Paul.Cotton@microsoft.com>, David Dorwin <ddorwin@google.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, Philippe Le Hégaret <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>
- Message-ID: <SN2PR03MB047275ED7A02FB88DB0616CA4C50@SN2PR03MB047.namprd03.prod.outlook.com>
General:
Most of the Google tests have been migrated, and all of those are now disabled in the repository. I ran the tests late yesterday, and am produced modified reports that show polyfill results IF the test fails for a given browser without the polyfill. If you see “Polyfill” in the report, it means just that. Polyfill results aren’t counted as a pass, and so show up in the failure reports as well.
Test reports:
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/less-than-2.html
- https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/complete-fails.html
Summary:
Total TypeError failures: 14 tests Instead of InvalidAccessError
Total Timeout issues: 17 tests Many on persistent session types or specific temporary session use cases
Total Polyfill passes: 5 tests On persistent session types
Total temporary playback with one pass: 11 tests Need to be fixed
Total error case tests: 8 tests Are these tests necessary for CR?
Total test scoring issues, not real problems: 14 tests One passing implementation because test is keySystem specific
Misc other failures: 11 tests
Total subtest failures: 80 tests
At Risk Based on Test:
1. Temporary sessions
2. Persistent-license sessions
3. Persistent-usage-record sessions
Questions :
1. When can temporary session type failures be fixed?
2. When will TypeError and promise rejection failures be fixed?
3. Are all error case tests required to prove the spec?
Details:
1. /encrypted-media/clearkey-generate-request-disallowed-input.html (7/7, 100.00%, 2.61% of total)
CH55 & FF52 Fail: TypeError expected, got InvalidAccessError (7 tests)
2. /encrypted-media/clearkey-keystatuses.html (1/1, 100.00%, 0.37% of total)
CH55 Pass, FF 52 Fail: assert_equals: expected 0 but got 2
3. /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
CH55 & FF%2 Pass with Polyfill – Known implementation issue, but include persistent sessions types that are not currently planned to be supported by all browsers (5 tests)
4. /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/clearkey-mp4-playback-temporary-multisession.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-expiration.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-expiration.html
CH55, FF52, ED14: Test Timeouts (7 tests)
5. /encrypted-media/drm-generate-request-disallowed-input.html (7/7, 100.00%, 2.61% of total)
CH55, ED14, FF52: Most failures expect TypeError, got InValidAccessError (7 tests)
6. /encrypted-media/drm-keystatuses-multiple-sessions.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-keystatuses.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
CH55, ED14, FF52: Mostly Timeout (7 tests)
7. /encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multikey.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multisession.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-two-videos.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-waitingforkey.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14 Timeout, FF52 Pass: All playback tests with MP4 content (11 tests)
8. /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14 Timeout, FF52 Fail: Error case, shouldn’t play with persistent license, but does (1 test)
9. /encrypted-media/idlharness.html (10/58, 17.24%, 3.73% of total)
CH55 & ED14 Fail: Throws TypeError instead of rejecting promise; FF52 Pass (9 tests)
CH55, ED14, FF52 Fail: MediaKeyMessageEvent.length expected 2, but got 1 (1 test)
10. /encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (4/58, 6.90%, 1.49% of total)
CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax checks/error cases (4 tests)
11. /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.37% of total)
CH55, ED14, FF52: Timeout (3 tests)
12. /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%, 0.37% of total)
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.37% of total)
CH55 & ED14: Timeout (2 tests)
FF52: Fail one test for sequence length of 9, 10 expected
FF52: Fail one test for Promise still outstanding at MediaKeys shutdown
13. /encrypted-media/drm-mp4-requestmediakeysystemaccess.html (17/72, 23.61%, 6.34% of total
CH55, ED14, FF52 Fail one or more RequestMediaKeySystemAccess syntax checks/error cases (3 tests)
ED14 only Pass: Test scoring issue, not valid single implementation feature (14 tests)
Jerry
Received on Tuesday, 4 October 2016 18:40:56 UTC