Re: [EME] Addressing Less Than 2 Passes tests

On Wed, Sep 14, 2016 at 12:50 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> I haven’t yet located a Chrome OS device to run tests on.  Does anyone on
> this thread have access?  If so, we might coordinate to include a json from
> it in our results.
>

I have initial results for these five tests. I'm looking into failures - it
looks like there may be a bug in at least one test. I'll send you results
this afternoon.
>
>
>
> I’ve confirmed ddorwin’s merge resolved issues in one test, and that a
> number of drm temporary scenarios are passing now that we have at least a
> temporary fix for license server issues.
>
>
>
> I’ll post an update in the morning, with or without Chrome OS results.
>
>
>
> Jerry
>
>
>
> *From:* Jerry Smith (WPT)
> *Sent:* Wednesday, September 14, 2016 11:14 AM
> *To:* 'David Dorwin' <ddorwin@google.com>; Chris Pearce <
> cpearce@mozilla.com>; Anthony Jones <ajones@mozilla.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <
> watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <
> public-html-media@w3.org> <public-html-media@w3.org>
> *Subject:* RE: [EME] Addressing Less Than 2 Passes tests
>
>
>
> We have traction now on the DRMToday license server issue, and it unblocks
> a number of previous timeout/failures.  I’ll post updated test status
> reports soon.
>
>
>
> I’ve been posting just CH55, FF51 and ED14 results because I can run them
> quickly on my desktop.  I will look into including Chrome OS results to
> address include the persistent test case passes you mention below, David.
>
>
>
> Jerry
>
>
>
> *From:* David Dorwin [mailto:ddorwin@google.com <ddorwin@google.com>]
> *Sent:* Tuesday, September 13, 2016 10:39 PM
> *To:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Chris Pearce <
> cpearce@mozilla.com>; Anthony Jones <ajones@mozilla.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <
> watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <
> public-html-media@w3.org> <public-html-media@w3.org>
> *Subject:* Re: [EME] Addressing Less Than 2 Passes tests
>
>
>
> +Anthony - Please see the requests for Chris below.
>
>
>
> ---------- Forwarded message ----------
> From: *David Dorwin* <ddorwin@google.com>
> Date: Tue, Sep 13, 2016 at 9:55 PM
> Subject: [EME] Addressing Less Than 2 Passes tests
> To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Chris Pearce <
> cpearce@mozilla.com>
> Cc: Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <
> watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "<
> public-html-media@w3.org>" <public-html-media@w3.org>
>
> [WAS: Updated EME Test Status]
>
>
>
> The following is an analysis of the less-than-2 results
> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
> published earlier today. There are specific requests for *Jerry* and
> *Chris* below, but we could use help from everyone.
>
>
>
> If you can help with the DRMtoday server issues or migrate the tests to
> use the new test account
> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us
> know ASAP as this is blocking all drm-* tests.
>
>
>
> Until the DRMtoday test server issue is resolved, we'll have to mostly
> ignore the drm-* tests since we do not have good results. However, I've
> commented on some that were failing in the test results before the today's
> update.
>
>
>
> In addition to the failures below, we may also see additional test
> failures as the Google/ tests are migrated and start running on Edge and
> Firefox as well exercising commercial Key Systems.
>
>
>
> *Google/**
>
> All of the test in Google/ are expected to potentially fail on other
> browsers until the tests are migrated. We need help migrating the remaining
> tests: https://github.com/w3c/web-platform-tests/issues/3583
> #issuecomment-243577488.
>
>
>
> (I merged an update to Google/encrypted-media-syntax.html that fixes the
> new failure reported in Chrome 55.)
>
>
>
> *idlharness.html*
>
>    - Three "interface: attribute" tests for the new event handler
>    attributes:
>
>
>    - Chrome passes these.
>
>
>    - We need another browser to implement the three new event handler
>       attributes. This is trivial, so while it should not block PR, it should be
>       easy to fix.
>
>
>    - *Jerry* and *Chris*, do you have plans to implement these?
>
>
>    - Nine "interface: operation" tests:
>
>
>    - Firefox passes these.
>       - The failures in Chrome are known
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>       caused by a Blink issue
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>       to EME.
>       - Edge also fails these tests. *Jerry*, do you know why?
>       - I think we can just note this in the test report.
>
> (While not an issue for PR since Edge and Firefox pass, for future
> reference, the six "existence and properties of interface prototype object"
> test failures in Chrome are also known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
> unrelated to EME.)
>
>
>
> *clearkey-*persistent**
>
> These eight tests (all complete-fails) are all related to persistent-*
> session types. No Clear Key implementation supports anything other than
> temporary sessions, which makes sense, and I don't expect this to change.
> Thus, these tests will not pass in the v1 timeframe.
>
>
>
> *clearkey-keystatuses.html*
>
> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
> to pass this test. It currently fails with:
>
> assert_equals: keystatus value for invalid key should be undefined (1)
> expected (undefined) undefined but got (string) "internal-error"
>
>
>
> *Chris*, please take a look.
>
>
>
> *drm-keystatuses.html*
>
> Mark wrote
> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
> :
>
> ... it appears that all three browsers are non-compliant:
>
>    - Chrome does not generate a keystatuseschange event after close() is
>    called (for the DRM case, it does for the ClearKey case)
>
>
>    - Firefox and Edge both have incorrect values in the keystatuses map
>    (in different ways).
>
>
>
> The Chrome issue is likely the same as this known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
>
> *Jerry* and *Chris*, please take a look.
>
>
>
> *drm-mp4-playback-temporary-events.html*
>
> Last time the tests were run, this was passing on Firefox, failing on
> Chrome, and timing out on Edge.
>
>
>
> This Chrome issue is known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
> investigation.
>
> *Jerry*, do you know why Edge was timing out?
>
>
>
> *drm-mp4-playback-temporary-waitingforkey.html*
>
> Last time the tests were run, this was timing out on all three browsers,
> but Mark landed a fix.
>
>
>
> *drm-mp4-playback-*persistent-license**
>
> "persistent-license" sessions are only supported on Chrome on Chrome OS,
> and I'm not aware of any plans to support them in other implementations
> before PR.
>
>
>
> At least one of these were failing the last step on Chrome OS due to a
> DRMtoday server issue.
>
>
>
> *mp4-playback-*persistent-usage-record**
>
> "persistent-usage-record" sessions are only supported in Edge.
>
>
>
> Last time the tests were run, none of the three tests were passing on Edge.
>
>
>
>
>
> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> I’ve posted updated results for EME tests:
>
>
>
> -          http://w3c.github.io/test-results/encrypted-media/all.html
>
> o   Test files: 70; Total subtests: 220
>
> -          http://w3c.github.io/test-results/encrypted-media/less-than-
> 2.html
>
> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
> Failure level: 64/220 (29.09%)
>
> -          http://w3c.github.io/test-results/encrypted-media/complete-
> fails.html
>
> o   Completely failed files: 44; Completely failed subtests: 29; Failure
> level: 29/220 (13.18%)
>
>
>
> Some comments about these results:
>
>
>
> 1.       They don’t filter for single valid test outcomes (either pass or
> fail), and timeouts as failures.  Most of the complete-fails are test
> timeouts.
>
> 2.       Test cases have only been partially (~25%) migrated to full drm
> from Clear Key versions contributed by Google.  The original Google tests
> are run if not migrated, and can be distinguished by “Google” in the test
> file path.
>
>
>
> Jerry
>
>
>
>
>