RE: [EME] Addressing Less Than 2 Passes tests

I haven’t yet located a Chrome OS device to run tests on.  Does anyone on this thread have access?  If so, we might coordinate to include a json from it in our results.

I’ve confirmed ddorwin’s merge resolved issues in one test, and that a number of drm temporary scenarios are passing now that we have at least a temporary fix for license server issues.

I’ll post an update in the morning, with or without Chrome OS results.

Jerry

From: Jerry Smith (WPT)
Sent: Wednesday, September 14, 2016 11:14 AM
To: 'David Dorwin' <ddorwin@google.com>; Chris Pearce <cpearce@mozilla.com>; Anthony Jones <ajones@mozilla.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <public-html-media@w3.org> <public-html-media@w3.org>
Subject: RE: [EME] Addressing Less Than 2 Passes tests

We have traction now on the DRMToday license server issue, and it unblocks a number of previous timeout/failures.  I’ll post updated test status reports soon.

I’ve been posting just CH55, FF51 and ED14 results because I can run them quickly on my desktop.  I will look into including Chrome OS results to address include the persistent test case passes you mention below, David.

Jerry

From: David Dorwin [mailto:ddorwin@google.com]
Sent: Tuesday, September 13, 2016 10:39 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>
Cc: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; <public-html-media@w3.org<mailto:public-html-media@w3.org>> <public-html-media@w3.org<mailto:public-html-media@w3.org>>
Subject: Re: [EME] Addressing Less Than 2 Passes tests

+Anthony - Please see the requests for Chris below.

---------- Forwarded message ----------
From: David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>
Date: Tue, Sep 13, 2016 at 9:55 PM
Subject: [EME] Addressing Less Than 2 Passes tests
To: "Jerry Smith (WPT)" <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>, Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>
Cc: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>, Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>, "<public-html-media@w3.org<mailto:public-html-media@w3.org>>" <public-html-media@w3.org<mailto:public-html-media@w3.org>>
[WAS: Updated EME Test Status]

The following is an analysis of the less-than-2 results<http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry published earlier today. There are specific requests for Jerry and Chris below, but we could use help from everyone.

If you can help with the DRMtoday server issues or migrate the tests to use the new test account<https://github.com/w3c/web-platform-tests/issues/3624>, please let us know ASAP as this is blocking all drm-* tests.

Until the DRMtoday test server issue is resolved, we'll have to mostly ignore the drm-* tests since we do not have good results. However, I've commented on some that were failing in the test results before the today's update.

In addition to the failures below, we may also see additional test failures as the Google/ tests are migrated and start running on Edge and Firefox as well exercising commercial Key Systems.

Google/*
All of the test in Google/ are expected to potentially fail on other browsers until the tests are migrated. We need help migrating the remaining tests: https://github.com/w3c/web-platform-tests/issues/3583#issuecomment-243577488.


(I merged an update to Google/encrypted-media-syntax.html that fixes the new failure reported in Chrome 55.)

idlharness.html

  *   Three "interface: attribute" tests for the new event handler attributes:

     *   Chrome passes these.

     *   We need another browser to implement the three new event handler attributes. This is trivial, so while it should not block PR, it should be easy to fix.

     *   Jerry and Chris, do you have plans to implement these?

  *   Nine "interface: operation" tests:

     *   Firefox passes these.
     *   The failures in Chrome are known<https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and caused by a Blink issue<https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated to EME.
     *   Edge also fails these tests. Jerry, do you know why?
     *   I think we can just note this in the test report.
(While not an issue for PR since Edge and Firefox pass, for future reference, the six "existence and properties of interface prototype object" test failures in Chrome are also known<https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and unrelated to EME.)

clearkey-*persistent*
These eight tests (all complete-fails) are all related to persistent-* session types. No Clear Key implementation supports anything other than temporary sessions, which makes sense, and I don't expect this to change. Thus, these tests will not pass in the v1 timeframe.

clearkey-keystatuses.html
Chrome passes. Since Edge does not implement Clear Key, we need Firefox to pass this test. It currently fails with:
assert_equals: keystatus value for invalid key should be undefined (1) expected (undefined) undefined but got (string) "internal-error"

Chris, please take a look.

drm-keystatuses.html
Mark wrote<https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>:
... it appears that all three browsers are non-compliant:

  *   Chrome does not generate a keystatuseschange event after close() is called (for the DRM case, it does for the ClearKey case)

  *   Firefox and Edge both have incorrect values in the keystatuses map (in different ways).

The Chrome issue is likely the same as this known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
Jerry and Chris, please take a look.

drm-mp4-playback-temporary-events.html
Last time the tests were run, this was passing on Firefox, failing on Chrome, and timing out on Edge.

This Chrome issue is known<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under investigation.
Jerry, do you know why Edge was timing out?

drm-mp4-playback-temporary-waitingforkey.html
Last time the tests were run, this was timing out on all three browsers, but Mark landed a fix.

drm-mp4-playback-*persistent-license*
"persistent-license" sessions are only supported on Chrome on Chrome OS, and I'm not aware of any plans to support them in other implementations before PR.

At least one of these were failing the last step on Chrome OS due to a DRMtoday server issue.

mp4-playback-*persistent-usage-record*
"persistent-usage-record" sessions are only supported in Edge.

Last time the tests were run, none of the three tests were passing on Edge.


On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
I’ve posted updated results for EME tests:


-          http://w3c.github.io/test-results/encrypted-media/all.html


o   Test files: 70; Total subtests: 220

-          http://w3c.github.io/test-results/encrypted-media/less-than-2.html


o   Test files without 2 passes: 44; Subtests without 2 passes: 64; Failure level: 64/220 (29.09%)

-          http://w3c.github.io/test-results/encrypted-media/complete-fails.html


o   Completely failed files: 44; Completely failed subtests: 29; Failure level: 29/220 (13.18%)

Some comments about these results:


1.       They don’t filter for single valid test outcomes (either pass or fail), and timeouts as failures.  Most of the complete-fails are test timeouts.

2.       Test cases have only been partially (~25%) migrated to full drm from Clear Key versions contributed by Google.  The original Google tests are run if not migrated, and can be distinguished by “Google” in the test file path.

Jerry