Re: [EME] Addressing Less Than 2 Passes tests

On Tue, Sep 20, 2016 at 12:39 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> David asked me some specific questions.  Generally, we are not able to
> update Edge to close implementation gaps anytime soon.  Some of the changes
> are minor, but dev resources are currently assigned on other tasks.
>
>
>
> Consolidating the questions then:
>
>
>
> *idlharness.html*
>
>    - Three "interface: attribute" tests for the new event handler
>    attributes:
>       - Chrome passes these.
>       - We need another browser to implement the three new event handler
>       attributes. This is trivial, so while it should not block PR, it should be
>       easy to fix.
>       - *Jerry* and *Chris*, do you have plans to implement these?
>          - *From Jerry:  We do plan to implement these, but not soon
>          enough to provide test help on EME.  I agree they are not difficult to
>          implement.*
>       - Nine "interface: operation" tests:
>       - Firefox passes these.
>       - The failures in Chrome are known
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>       caused by a Blink issue
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>       to EME.
>       - Edge also fails these tests. *Jerry*, do you know why?
>          - *From Jerry:  *I believe these reflect Edge bugs, not test
>          issues:
>             - Navigator interface: operation requestMediaKeySystemAccess(DOMString,[object
>             Object]) – Property length 2, got 0.
>             - HTMLMediaElement interface: operation
>             setMediaKeys(MediaKeys) – Property length 1, got 0.
>             - MediaKeySystemAccess interface: operation createMediaKeys()
>             – Did not reject promise as expected.
>             - MediaKeys interface: operation setServerCertificate(BufferSource)
>             – Property length expected 1, got 0.
>             - MediaKeySession interface: operation
>             generateRequest(DOMString,BufferSource) – Property length 2,
>             got 0.
>             - MediaKeySession interface: operation load(DOMString) –
>             Property length 1, got 0.
>             - MediaKeySession interface: operation update(BufferSource) –
>             Property length 1, got 0.
>             - MediaKeySession interface: operation close()– Did not
>             reject promise as expected.
>             - MediaKeySession interface: operation remove() – Did not
>             reject promise as expected.
>
> *drm-keystatuses.html*
>
> -          Mark wrote
> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
> :
>
> o   ... it appears that all three browsers are non-compliant:
>
> o   Chrome does not generate a keystatuseschange event after close() is
> called (for the DRM case, it does for the ClearKey case)
>
> o   Firefox and Edge both have incorrect values in the keystatuses map
> (in different ways).
>
> -          The Chrome issue is likely the same as this known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
>
> -          *Jerry* and *Chris*, please take a look.
>
> o   *From Jerry:*  Edge fails because it currently does not support the
> iterator method.  This is a known bug.
>

​There are other bugs with Edge, which I have filed as bugs (I can dig up
the numbers, but I provided them previously). Specifically, the keyid and
status values are provided to the forEach callback in the wrong order and
the keyid value is incorrect (various byte swaps have been applied).​


>
>
> *drm-mp4-playback-temporary-events.html*
>
> -          Last time the tests were run, this was passing on Firefox,
> failing on Chrome, and timing out on Edge.
>
> -          This Chrome issue is known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
> investigation.
>
> -          *Jerry*, do you know why Edge was timing out?
>
> o   *From Jerry:*  Timesout on MediaKeySession.keystatuses because Edge
> is returning an ArrayBuffer instead of a key status string.  I’m not sure
> why this isn’t being recorded as a fail.
>
​Same issue as above, forEach arguments are in the wrong order.​

>
>
> Jerry
>
>
>
> *From:* David Dorwin [mailto:ddorwin@google.com]
> *Sent:* Tuesday, September 13, 2016 9:56 PM
> *To:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Chris Pearce <
> cpearce@mozilla.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <
> watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <
> public-html-media@w3.org> <public-html-media@w3.org>
> *Subject:* [EME] Addressing Less Than 2 Passes tests
>
>
>
> [WAS: Updated EME Test Status]
>
>
>
> The following is an analysis of the less-than-2 results
> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
> published earlier today. There are specific requests for *Jerry* and
> *Chris* below, but we could use help from everyone.
>
>
>
> If you can help with the DRMtoday server issues or migrate the tests to
> use the new test account
> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us
> know ASAP as this is blocking all drm-* tests.
>
>
>
> Until the DRMtoday test server issue is resolved, we'll have to mostly
> ignore the drm-* tests since we do not have good results. However, I've
> commented on some that were failing in the test results before the today's
> update.
>
>
>
> In addition to the failures below, we may also see additional test
> failures as the Google/ tests are migrated and start running on Edge and
> Firefox as well exercising commercial Key Systems.
>
>
>
> *Google/**
>
> All of the test in Google/ are expected to potentially fail on other
> browsers until the tests are migrated. We need help migrating the remaining
> tests: https://github.com/w3c/web-platform-tests/issues/
> 3583#issuecomment-243577488.
>
>
>
> (I merged an update to Google/encrypted-media-syntax.html that fixes the
> new failure reported in Chrome 55.)
>
>
>
> *idlharness.html*
>
>    - Three "interface: attribute" tests for the new event handler
>    attributes:
>
>
>    - Chrome passes these.
>
>
>    - We need another browser to implement the three new event handler
>       attributes. This is trivial, so while it should not block PR, it should be
>       easy to fix.
>
>
>    - *Jerry* and *Chris*, do you have plans to implement these?
>
>
>    - Nine "interface: operation" tests:
>
>
>    - Firefox passes these.
>       - The failures in Chrome are known
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>       caused by a Blink issue
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>       to EME.
>       - Edge also fails these tests. *Jerry*, do you know why?
>       - I think we can just note this in the test report.
>
> (While not an issue for PR since Edge and Firefox pass, for future
> reference, the six "existence and properties of interface prototype object"
> test failures in Chrome are also known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
> unrelated to EME.)
>
>
>
> *clearkey-*persistent**
>
> These eight tests (all complete-fails) are all related to persistent-*
> session types. No Clear Key implementation supports anything other than
> temporary sessions, which makes sense, and I don't expect this to change.
> Thus, these tests will not pass in the v1 timeframe.
>
>
>
> *clearkey-keystatuses.html*
>
> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
> to pass this test. It currently fails with:
>
> assert_equals: keystatus value for invalid key should be undefined (1)
> expected (undefined) undefined but got (string) "internal-error"
>
>
>
> *Chris*, please take a look.
>
>
>
> *drm-keystatuses.html*
>
> Mark wrote
> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
> :
>
> ... it appears that all three browsers are non-compliant:
>
>    - Chrome does not generate a keystatuseschange event after close() is
>    called (for the DRM case, it does for the ClearKey case)
>
>
>    - Firefox and Edge both have incorrect values in the keystatuses map
>    (in different ways).
>
>
>
> The Chrome issue is likely the same as this known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
>
> *Jerry* and *Chris*, please take a look.
>
>
>
> *drm-mp4-playback-temporary-events.html*
>
> Last time the tests were run, this was passing on Firefox, failing on
> Chrome, and timing out on Edge.
>
>
>
> This Chrome issue is known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
> investigation.
>
> *Jerry*, do you know why Edge was timing out?
>
>
>
> *drm-mp4-playback-temporary-waitingforkey.html*
>
> Last time the tests were run, this was timing out on all three browsers,
> but Mark landed a fix.
>
>
>
> *drm-mp4-playback-*persistent-license**
>
> "persistent-license" sessions are only supported on Chrome on Chrome OS,
> and I'm not aware of any plans to support them in other implementations
> before PR.
>
>
>
> At least one of these were failing the last step on Chrome OS due to a
> DRMtoday server issue.
>
>
>
> *mp4-playback-*persistent-usage-record**
>
> "persistent-usage-record" sessions are only supported in Edge.
>
>
>
> Last time the tests were run, none of the three tests were passing on Edge.
>
>
>
>
>
> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> I’ve posted updated results for EME tests:
>
>
>
> -          http://w3c.github.io/test-results/encrypted-media/all.html
>
> o   Test files: 70; Total subtests: 220
>
> -          http://w3c.github.io/test-results/encrypted-media/less-
> than-2.html
>
> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
> Failure level: 64/220 (29.09%)
>
> -          http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> o   Completely failed files: 44; Completely failed subtests: 29; Failure
> level: 29/220 (13.18%)
>
>
>
> Some comments about these results:
>
>
>
> 1.       They don’t filter for single valid test outcomes (either pass or
> fail), and timeouts as failures.  Most of the complete-fails are test
> timeouts.
>
> 2.       Test cases have only been partially (~25%) migrated to full drm
> from Clear Key versions contributed by Google.  The original Google tests
> are run if not migrated, and can be distinguished by “Google” in the test
> file path.
>
>
>
> Jerry
>
>
>

Received on Monday, 19 September 2016 23:43:45 UTC