Re: [EME] Addressing Less Than 2 Passes tests

+Anthony - Please see the requests for Chris below.

---------- Forwarded message ----------
From: David Dorwin <ddorwin@google.com>
Date: Tue, Sep 13, 2016 at 9:55 PM
Subject: [EME] Addressing Less Than 2 Passes tests
To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Chris Pearce <
cpearce@mozilla.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <
watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "<
public-html-media@w3.org>" <public-html-media@w3.org>


[WAS: Updated EME Test Status]

The following is an analysis of the less-than-2 results
<http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
published earlier today. There are specific requests for *Jerry* and
*Chris* below,
but we could use help from everyone.

If you can help with the DRMtoday server issues or migrate the tests to use
the new test account <https://github.com/w3c/web-platform-tests/issues/3624>,
please let us know ASAP as this is blocking all drm-* tests.

Until the DRMtoday test server issue is resolved, we'll have to mostly
ignore the drm-* tests since we do not have good results. However, I've
commented on some that were failing in the test results before the today's
update.

In addition to the failures below, we may also see additional test failures
as the Google/ tests are migrated and start running on Edge and Firefox as
well exercising commercial Key Systems.

*Google/**
All of the test in Google/ are expected to potentially fail on other
browsers until the tests are migrated. We need help migrating the remaining
tests: https://github.com/w3c/web-platform-tests/issues/3583
#issuecomment-243577488.

(I merged an update to Google/encrypted-media-syntax.html that fixes the
new failure reported in Chrome 55.)

*idlharness.html*

   - Three "interface: attribute" tests for the new event handler
   attributes:
      - Chrome passes these.
      - We need another browser to implement the three new event handler
      attributes. This is trivial, so while it should not block PR, it
should be
      easy to fix.
      - *Jerry* and *Chris*, do you have plans to implement these?
      - Nine "interface: operation" tests:
      - Firefox passes these.
      - The failures in Chrome are known
      <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
      caused by a Blink issue
      <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
      to EME.
      - Edge also fails these tests. *Jerry*, do you know why?
      - I think we can just note this in the test report.

(While not an issue for PR since Edge and Firefox pass, for future
reference, the six "existence and properties of interface prototype object"
test failures in Chrome are also known
<https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
unrelated to EME.)

*clearkey-*persistent**
These eight tests (all complete-fails) are all related to persistent-*
session types. No Clear Key implementation supports anything other than
temporary sessions, which makes sense, and I don't expect this to change.
Thus, these tests will not pass in the v1 timeframe.

*clearkey-keystatuses.html*
Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
to pass this test. It currently fails with:

> assert_equals: keystatus value for invalid key should be undefined (1)
> expected (undefined) undefined but got (string) "internal-error"
>

*Chris*, please take a look.

*drm-keystatuses.html*
Mark wrote
<https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
:

> ... it appears that all three browsers are non-compliant:
>
>    - Chrome does not generate a keystatuseschange event after close() is
>    called (for the DRM case, it does for the ClearKey case)
>
>
>    - Firefox and Edge both have incorrect values in the keystatuses map
>    (in different ways).
>
>
The Chrome issue is likely the same as this known
<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
*Jerry* and *Chris*, please take a look.

*drm-mp4-playback-temporary-events.html*
Last time the tests were run, this was passing on Firefox, failing on
Chrome, and timing out on Edge.

This Chrome issue is known
<https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
investigation.
*Jerry*, do you know why Edge was timing out?

*drm-mp4-playback-temporary-waitingforkey.html*
Last time the tests were run, this was timing out on all three browsers,
but Mark landed a fix.

*drm-mp4-playback-*persistent-license**
"persistent-license" sessions are only supported on Chrome on Chrome OS,
and I'm not aware of any plans to support them in other implementations
before PR.

At least one of these were failing the last step on Chrome OS due to a
DRMtoday server issue.

*mp4-playback-*persistent-usage-record**
"persistent-usage-record" sessions are only supported in Edge.

Last time the tests were run, none of the three tests were passing on Edge.


On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> I’ve posted updated results for EME tests:
>
>
>
> -          http://w3c.github.io/test-results/encrypted-media/all.html
>
> o   Test files: 70; Total subtests: 220
>
> -          http://w3c.github.io/test-results/encrypted-media/less-than-
> 2.html
>
> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
> Failure level: 64/220 (29.09%)
>
> -          http://w3c.github.io/test-results/encrypted-media/complete-f
> ails.html
>
> o   Completely failed files: 44; Completely failed subtests: 29; Failure
> level: 29/220 (13.18%)
>
>
>
> Some comments about these results:
>
>
>
> 1.       They don’t filter for single valid test outcomes (either pass or
> fail), and timeouts as failures.  Most of the complete-fails are test
> timeouts.
>
> 2.       Test cases have only been partially (~25%) migrated to full drm
> from Clear Key versions contributed by Google.  The original Google tests
> are run if not migrated, and can be distinguished by “Google” in the test
> file path.
>
>
>
> Jerry
>