RE: [EME] Addressing Less Than 2 Passes tests from Jerry Smith (WPT) on 2016-09-20 (public-html-media@w3.org from September 2016)

From: Jerry Smith (WPT) <jdsmith@microsoft.com>
Date: Tue, 20 Sep 2016 20:58:40 +0000
To: Mark Watson <watsonm@netflix.com>
CC: David Dorwin <ddorwin@google.com>, Chris Pearce <cpearce@mozilla.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Matt Wolenetz <wolenetz@google.com>, "<public-html-media@w3.org>" <public-html-media@w3.org>
Message-ID: <SN2PR03MB0474902D0C15349B3D4A13DA4F70@SN2PR03MB047.namprd03.prod.outlook.com>
Edge public bugs that document the failures below are:

EME Compliance: idlharness test failures:
-   https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8997139/


EME Compliance: keyStatuses:
-   https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8535903/


EME Compliance: persistent-usage-record:
-   https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8536237/


Plus this one:

EME Compliance: requestMediaKeySystemAccess initData types:
-   https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8536037/



Mark:  The bugs you entered are included on this list.

Jerry

From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Monday, September 19, 2016 4:43 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: David Dorwin <ddorwin@google.com>; Chris Pearce <cpearce@mozilla.com>; Paul Cotton <Paul.Cotton@microsoft.com>; Matt Wolenetz <wolenetz@google.com>; <public-html-media@w3.org> <public-html-media@w3.org>
Subject: Re: [EME] Addressing Less Than 2 Passes tests



On Tue, Sep 20, 2016 at 12:39 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
David asked me some specific questions.  Generally, we are not able to update Edge to close implementation gaps anytime soon.  Some of the changes are minor, but dev resources are currently assigned on other tasks.

Consolidating the questions then:

idlharness.html

  *   Three "interface: attribute" tests for the new event handler attributes:

     *   Chrome passes these.
     *   We need another browser to implement the three new event handler attributes. This is trivial, so while it should not block PR, it should be easy to fix.
     *   Jerry and Chris, do you have plans to implement these?

        *   From Jerry:  We do plan to implement these, but not soon enough to provide test help on EME.  I agree they are not difficult to implement.

  *   Nine "interface: operation" tests:

     *   Firefox passes these.
     *   The failures in Chrome are known<https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and caused by a Blink issue<https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated to EME.
     *   Edge also fails these tests. Jerry, do you know why?

        *   From Jerry:  I believe these reflect Edge bugs, not test issues:

           *   Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object Object]) – Property length 2, got 0.
           *   HTMLMediaElement interface: operation setMediaKeys(MediaKeys) – Property length 1, got 0.
           *   MediaKeySystemAccess interface: operation createMediaKeys() – Did not reject promise as expected.
           *   MediaKeys interface: operation setServerCertificate(BufferSource) – Property length expected 1, got 0.
           *   MediaKeySession interface: operation generateRequest(DOMString,BufferSource) – Property length 2, got 0.
           *   MediaKeySession interface: operation load(DOMString) – Property length 1, got 0.
           *   MediaKeySession interface: operation update(BufferSource) – Property length 1, got 0.
           *   MediaKeySession interface: operation close()– Did not reject promise as expected.
           *   MediaKeySession interface: operation remove() – Did not reject promise as expected.
drm-keystatuses.html

-          Mark wrote<https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>:

o   ... it appears that all three browsers are non-compliant:
o   Chrome does not generate a keystatuseschange event after close() is called (for the DRM case, it does for the ClearKey case)
o   Firefox and Edge both have incorrect values in the keystatuses map (in different ways).

-          The Chrome issue is likely the same as this known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.

-          Jerry and Chris, please take a look.

o   From Jerry:  Edge fails because it currently does not support the iterator method.  This is a known bug.

There are other bugs with Edge, which I have filed as bugs (I can dig up the numbers, but I provided them previously). Specifically, the keyid and status values are provided to the forEach callback in the wrong order and the keyid value is incorrect (various byte swaps have been applied).


drm-mp4-playback-temporary-events.html

-          Last time the tests were run, this was passing on Firefox, failing on Chrome, and timing out on Edge.

-          This Chrome issue is known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under investigation.

-          Jerry, do you know why Edge was timing out?

o   From Jerry:  Timesout on MediaKeySession.keystatuses because Edge is returning an ArrayBuffer instead of a key status string.  I’m not sure why this isn’t being recorded as a fail.
Same issue as above, forEach arguments are in the wrong order.

Jerry

From: David Dorwin [mailto:ddorwin@google.com<mailto:ddorwin@google.com>]
Sent: Tuesday, September 13, 2016 9:56 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>
Cc: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; <public-html-media@w3.org<mailto:public-html-media@w3.org>> <public-html-media@w3.org<mailto:public-html-media@w3.org>>
Subject: [EME] Addressing Less Than 2 Passes tests

[WAS: Updated EME Test Status]

The following is an analysis of the less-than-2 results<http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry published earlier today. There are specific requests for Jerry and Chris below, but we could use help from everyone.

If you can help with the DRMtoday server issues or migrate the tests to use the new test account<https://github.com/w3c/web-platform-tests/issues/3624>, please let us know ASAP as this is blocking all drm-* tests.

Until the DRMtoday test server issue is resolved, we'll have to mostly ignore the drm-* tests since we do not have good results. However, I've commented on some that were failing in the test results before the today's update.

In addition to the failures below, we may also see additional test failures as the Google/ tests are migrated and start running on Edge and Firefox as well exercising commercial Key Systems.

Google/*
All of the test in Google/ are expected to potentially fail on other browsers until the tests are migrated. We need help migrating the remaining tests: https://github.com/w3c/web-platform-tests/issues/3583#issuecomment-243577488.


(I merged an update to Google/encrypted-media-syntax.html that fixes the new failure reported in Chrome 55.)

idlharness.html

  *   Three "interface: attribute" tests for the new event handler attributes:

     *   Chrome passes these.

     *   We need another browser to implement the three new event handler attributes. This is trivial, so while it should not block PR, it should be easy to fix.

     *   Jerry and Chris, do you have plans to implement these?

  *   Nine "interface: operation" tests:

     *   Firefox passes these.
     *   The failures in Chrome are known<https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and caused by a Blink issue<https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated to EME.
     *   Edge also fails these tests. Jerry, do you know why?
     *   I think we can just note this in the test report.
(While not an issue for PR since Edge and Firefox pass, for future reference, the six "existence and properties of interface prototype object" test failures in Chrome are also known<https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and unrelated to EME.)

clearkey-*persistent*
These eight tests (all complete-fails) are all related to persistent-* session types. No Clear Key implementation supports anything other than temporary sessions, which makes sense, and I don't expect this to change. Thus, these tests will not pass in the v1 timeframe.

clearkey-keystatuses.html
Chrome passes. Since Edge does not implement Clear Key, we need Firefox to pass this test. It currently fails with:
assert_equals: keystatus value for invalid key should be undefined (1) expected (undefined) undefined but got (string) "internal-error"

Chris, please take a look.

drm-keystatuses.html
Mark wrote<https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>:
... it appears that all three browsers are non-compliant:

  *   Chrome does not generate a keystatuseschange event after close() is called (for the DRM case, it does for the ClearKey case)

  *   Firefox and Edge both have incorrect values in the keystatuses map (in different ways).

The Chrome issue is likely the same as this known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
Jerry and Chris, please take a look.

drm-mp4-playback-temporary-events.html
Last time the tests were run, this was passing on Firefox, failing on Chrome, and timing out on Edge.

This Chrome issue is known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under investigation.
Jerry, do you know why Edge was timing out?

drm-mp4-playback-temporary-waitingforkey.html
Last time the tests were run, this was timing out on all three browsers, but Mark landed a fix.

drm-mp4-playback-*persistent-license*
"persistent-license" sessions are only supported on Chrome on Chrome OS, and I'm not aware of any plans to support them in other implementations before PR.

At least one of these were failing the last step on Chrome OS due to a DRMtoday server issue.

mp4-playback-*persistent-usage-record*
"persistent-usage-record" sessions are only supported in Edge.

Last time the tests were run, none of the three tests were passing on Edge.


On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
I’ve posted updated results for EME tests:


-          http://w3c.github.io/test-results/encrypted-media/all.html


o   Test files: 70; Total subtests: 220

-          http://w3c.github.io/test-results/encrypted-media/less-than-2.html


o   Test files without 2 passes: 44; Subtests without 2 passes: 64; Failure level: 64/220 (29.09%)

-          http://w3c.github.io/test-results/encrypted-media/complete-fails.html


o   Completely failed files: 44; Completely failed subtests: 29; Failure level: 29/220 (13.18%)

Some comments about these results:


1.       They don’t filter for single valid test outcomes (either pass or fail), and timeouts as failures.  Most of the complete-fails are test timeouts.

2.       Test cases have only been partially (~25%) migrated to full drm from Clear Key versions contributed by Google.  The original Google tests are run if not migrated, and can be distinguished by “Google” in the test file path.

Jerry
Received on Tuesday, 20 September 2016 20:59:12 UTC