W3C home > Mailing lists > Public > public-html-media@w3.org > October 2016

EME Test Status Update - 10-19-16

From: Jerry Smith (WPT) <jdsmith@microsoft.com>
Date: Wed, 19 Oct 2016 21:29:31 +0000
To: Paul Cotton <Paul.Cotton@microsoft.com>, "'public-html-media@w3.org'" <public-html-media@w3.org>
Message-ID: <BY2PR03MB041686741FE0F24DB30B15CA4D20@BY2PR03MB041.namprd03.prod.outlook.com>
I've updated the test status reports to reflect recent fixes.  More are expected in the next few days.


*       http://w3c.github.io/test-results/encrypted-media/all.html

o   Test files: 97; Total subtests: 306

*       http://w3c.github.io/test-results/encrypted-media/complete-fails.html

o   Completely failed files: 22; Completely failed subtests: 22; Failure level: 22/306 (7.19%)

*       http://w3c.github.io/test-results/encrypted-media/less-than-2.html

o   Test files without 2 passes: 33; Subtests without 2 passes: 47; Failure level: 47/306 (15.36%)
We discussed in the call on Tuesday treating some categories of test issues as non-blocking bugs.  There are in fact few issues currently that are viewed as feature failures on our tests.  I did a sort through these and am slightly off on the total tests.  I am out tomorrow, so am posting this as an interim status for comments, despite this.

In the category we consider non-blocking bugs are these:

Errors (considered bugs, not blocking)                                                                         23
Clear Key persistent-license (not planned, may make non-normative)                   3
Clear Key persistent-usage-record (not planned, no impact)                             3
Clear Key transitions between clear and encrypted (bugs, not blocking)                      2

Not Blocking Tests Total:                                                                                               31

In the category we are treating as implementation issues currently are these:

Persistent-license (fixes in work)                                                                                   5
Persistent-usage-record (no implementations in test, at risk)                                  3
Temporary session edge cases (bugs, not block, fixes in work)                                3
Temporary session readyState (issues under discussion, at risk)                                     2

In work or at risk Total:                                                                                                 13

Lists of the individual tests sorted into these groups are provided below.  Comments on the groupings are welcome.

Jerry


Error handling (considered bugs, not blocking) - 23 Failures
/encrypted-media/clearkey-mp4-syntax-mediakeysession.html (2/7, 28.57%, 0.65% of total)
org.w3.clearkey test MediaKeySession load() exceptions.
org.w3.clearkey test if MediaKeySession generateRequest() resolves for various sessions
/encrypted-media/clearkey-update-non-ascii-input.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey test handling of non-ASCII responses for update()
/encrypted-media/drm-generate-request-disallowed-input.html (3/7, 42.86%, 0.98% of total)
drm, temporary, keyids, initData longer than 64Kb characters
drm, temporary, keyids, invalid initdata (too short key ID)
drm, temporary, keyids, invalid initdata (too long key ID)
/encrypted-media/drm-mp4-syntax-mediakeysession.html (2/7, 28.57%, 0.65% of total)
drm test MediaKeySession load() exceptions.
drm test if MediaKeySession generateRequest() resolves for various sessions
/encrypted-media/idlharness.html (10/58, 17.24%, 3.27% of total)
Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object Object])
HTMLMediaElement interface: operation setMediaKeys(MediaKeys)
MediaKeySystemAccess interface: operation createMediaKeys()
MediaKeys interface: operation setServerCertificate(BufferSource)
MediaKeySession interface: operation generateRequest(DOMString,BufferSource)
MediaKeySession interface: operation load(DOMString)
MediaKeySession interface: operation update(BufferSource)
MediaKeySession interface: operation close()
MediaKeySession interface: operation remove()
MediaKeyMessageEvent interface object length
/encrypted-media/clearkey-mp4-requestmediakeysystemaccess.html (1/58, 1.72%, 0.33% of total)
Trailing space in contentType
/encrypted-media/drm-mp4-requestmediakeysystemaccess.html (2/58, 3.45%, 0.65% of total)
drm, requestMediaKeySystemAccess: Unrecognized foo with mp4 ('video/mp4; foo="bar"') should result in NotSupportedError
drm, requestMediaKeySystemAccess: Unrecognized foo with codecs ('video/mp4;codecs="avc1.4d401e"; foo="bar"') should result in NotSupportedError
/encrypted-media/clearkey-mp4-update-disallowed-input.html (1/1, 100.00%, 0.33% of total)
update() with invalid response (longer than 64Kb characters) should fail.
/encrypted-media/drm-invalid-license.html (1/1, 100.00%, 0.33% of total)
Update with invalid Clear Key license

Clear key persistent license tests (not planned to be implemented, may make informative note) - 3 Failures
/encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback

Clear key persistent-usage-case (not planned to be implemented) - 3 Failures
/encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, check events
/encrypted-media/clearkey-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4playback
/encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new window

Clear key transitions between clear and encrypted content (considered bugs, not blocking) - 2 Failures
/encrypted-media/clearkey-mp4-playback-temporary-clear-encrypted.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, temporary, mp4, playback, single key, clear then encrypted content
/encrypted-media/clearkey-mp4-playback-temporary-encrypted-clear.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, temporary, mp4, playback, single key, encrypted then clear content

Persistent-license failures - 5 Failures
/encrypted-media/drm-mp4-playback-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, destroy and acknowledge
/encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, check events
/encrypted-media/drm-mp4-playback-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4playback
/encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback and destroy
/encrypted-media/drm-mp4-playback-retrieve-persistent-license.html (1/1, 100.00%, 0.33% of total)
drm, persistent-license, mp4, playback, retrieve, playback

Persistent-usage-record failures - 3 Failures
/encrypted-media/drm-mp4-playback-persistent-usage-record-events.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, check events
/encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4, playback, retrieve in new window
/encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1, 100.00%, 0.33% of total)
drm, persistent-usage-record, mp4playback

Temporary session failures - 3 Failures
/encrypted-media/drm-mp4-playback-temporary-expired.html (1/1, 100.00%, 0.33% of total)
drm, temporary, mp4, expired license
/encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of total)
drm, cannot load persistent license into temporary session
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential

Temporary session readyState failures (listed at risk, under discussion) - 2 Failures
/encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
org.w3.clearkey, successful playback, temporary, mp4, multiple keys, sequential, readyState
/encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html (1/1, 100.00%, 0.33% of total)
drm, successful playback, temporary, mp4, multiple keys, sequential, readyState
Received on Wednesday, 19 October 2016 21:30:16 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 19 October 2016 21:30:17 UTC