Re: [EME] Addressing Less Than 2 Passes tests

I provided results to Jerry. 2 tests pass; 3 fail (timeout) near the end
due to server errors (bug
<https://github.com/w3c/web-platform-tests/issues/3718>). Mark, please
confirm that those are the same issues you saw and not something new or
that might be a test bug.

One of those three tests fails instead of timing out due to a test bug. PR
#3717 <https://github.com/w3c/web-platform-tests/pull/3717> fixes that.

On Wed, Sep 14, 2016 at 3:14 PM, Mark Watson <watsonm@netflix.com> wrote:

> I have a ChromeOS device, but I have not heard that the server bug for
> processing the record of license destruction has been fixed yet.
>
> ...Mark
>
> Sent from my iPhone
>
> On Sep 14, 2016, at 1:17 PM, David Dorwin <ddorwin@google.com> wrote:
>
>
> On Wed, Sep 14, 2016 at 12:50 PM, Jerry Smith (WPT) <jdsmith@microsoft.com
> > wrote:
>
>> I haven’t yet located a Chrome OS device to run tests on.  Does anyone on
>> this thread have access?  If so, we might coordinate to include a json from
>> it in our results.
>>
>
> I have initial results for these five tests. I'm looking into failures -
> it looks like there may be a bug in at least one test. I'll send you
> results this afternoon.
>>
>>
>>
>> I’ve confirmed ddorwin’s merge resolved issues in one test, and that a
>> number of drm temporary scenarios are passing now that we have at least a
>> temporary fix for license server issues.
>>
>>
>>
>> I’ll post an update in the morning, with or without Chrome OS results.
>>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* Jerry Smith (WPT)
>> *Sent:* Wednesday, September 14, 2016 11:14 AM
>> *To:* 'David Dorwin' <ddorwin@google.com>; Chris Pearce <
>> cpearce@mozilla.com>; Anthony Jones <ajones@mozilla.com>
>> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <
>> watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <
>> public-html-media@w3.org> <public-html-media@w3.org>
>> *Subject:* RE: [EME] Addressing Less Than 2 Passes tests
>>
>>
>>
>> We have traction now on the DRMToday license server issue, and it
>> unblocks a number of previous timeout/failures.  I’ll post updated test
>> status reports soon.
>>
>>
>>
>> I’ve been posting just CH55, FF51 and ED14 results because I can run them
>> quickly on my desktop.  I will look into including Chrome OS results to
>> address include the persistent test case passes you mention below, David.
>>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* David Dorwin [mailto:ddorwin@google.com <ddorwin@google.com>]
>> *Sent:* Tuesday, September 13, 2016 10:39 PM
>> *To:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Chris Pearce <
>> cpearce@mozilla.com>; Anthony Jones <ajones@mozilla.com>
>> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; Mark Watson <
>> watsonm@netflix.com>; Matt Wolenetz <wolenetz@google.com>; <
>> public-html-media@w3.org> <public-html-media@w3.org>
>> *Subject:* Re: [EME] Addressing Less Than 2 Passes tests
>>
>>
>>
>> +Anthony - Please see the requests for Chris below.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: *David Dorwin* <ddorwin@google.com>
>> Date: Tue, Sep 13, 2016 at 9:55 PM
>> Subject: [EME] Addressing Less Than 2 Passes tests
>> To: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Chris Pearce <
>> cpearce@mozilla.com>
>> Cc: Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <
>> watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "<
>> public-html-media@w3.org>" <public-html-media@w3.org>
>>
>> [WAS: Updated EME Test Status]
>>
>>
>>
>> The following is an analysis of the less-than-2 results
>> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
>> published earlier today. There are specific requests for *Jerry* and
>> *Chris* below, but we could use help from everyone.
>>
>>
>>
>> If you can help with the DRMtoday server issues or migrate the tests to
>> use the new test account
>> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us
>> know ASAP as this is blocking all drm-* tests.
>>
>>
>>
>> Until the DRMtoday test server issue is resolved, we'll have to mostly
>> ignore the drm-* tests since we do not have good results. However, I've
>> commented on some that were failing in the test results before the today's
>> update.
>>
>>
>>
>> In addition to the failures below, we may also see additional test
>> failures as the Google/ tests are migrated and start running on Edge and
>> Firefox as well exercising commercial Key Systems.
>>
>>
>>
>> *Google/**
>>
>> All of the test in Google/ are expected to potentially fail on other
>> browsers until the tests are migrated. We need help migrating the remaining
>> tests: https://github.com/w3c/web-platform-tests/issues/3583
>> #issuecomment-243577488.
>>
>>
>>
>> (I merged an update to Google/encrypted-media-syntax.html that fixes the
>> new failure reported in Chrome 55.)
>>
>>
>>
>> *idlharness.html*
>>
>>    - Three "interface: attribute" tests for the new event handler
>>    attributes:
>>
>>
>>    - Chrome passes these.
>>
>>
>>    - We need another browser to implement the three new event handler
>>       attributes. This is trivial, so while it should not block PR, it should be
>>       easy to fix.
>>
>>
>>    - *Jerry* and *Chris*, do you have plans to implement these?
>>
>>
>>    - Nine "interface: operation" tests:
>>
>>
>>    - Firefox passes these.
>>       - The failures in Chrome are known
>>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>>       caused by a Blink issue
>>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>>       to EME.
>>       - Edge also fails these tests. *Jerry*, do you know why?
>>       - I think we can just note this in the test report.
>>
>> (While not an issue for PR since Edge and Firefox pass, for future
>> reference, the six "existence and properties of interface prototype object"
>> test failures in Chrome are also known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
>> unrelated to EME.)
>>
>>
>>
>> *clearkey-*persistent**
>>
>> These eight tests (all complete-fails) are all related to persistent-*
>> session types. No Clear Key implementation supports anything other than
>> temporary sessions, which makes sense, and I don't expect this to change.
>> Thus, these tests will not pass in the v1 timeframe.
>>
>>
>>
>> *clearkey-keystatuses.html*
>>
>> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
>> to pass this test. It currently fails with:
>>
>> assert_equals: keystatus value for invalid key should be undefined (1)
>> expected (undefined) undefined but got (string) "internal-error"
>>
>>
>>
>> *Chris*, please take a look.
>>
>>
>>
>> *drm-keystatuses.html*
>>
>> Mark wrote
>> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
>> :
>>
>> ... it appears that all three browsers are non-compliant:
>>
>>    - Chrome does not generate a keystatuseschange event after close() is
>>    called (for the DRM case, it does for the ClearKey case)
>>
>>
>>    - Firefox and Edge both have incorrect values in the keystatuses map
>>    (in different ways).
>>
>>
>>
>> The Chrome issue is likely the same as this known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
>>
>> *Jerry* and *Chris*, please take a look.
>>
>>
>>
>> *drm-mp4-playback-temporary-events.html*
>>
>> Last time the tests were run, this was passing on Firefox, failing on
>> Chrome, and timing out on Edge.
>>
>>
>>
>> This Chrome issue is known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
>> investigation.
>>
>> *Jerry*, do you know why Edge was timing out?
>>
>>
>>
>> *drm-mp4-playback-temporary-waitingforkey.html*
>>
>> Last time the tests were run, this was timing out on all three browsers,
>> but Mark landed a fix.
>>
>>
>>
>> *drm-mp4-playback-*persistent-license**
>>
>> "persistent-license" sessions are only supported on Chrome on Chrome OS,
>> and I'm not aware of any plans to support them in other implementations
>> before PR.
>>
>>
>>
>> At least one of these were failing the last step on Chrome OS due to a
>> DRMtoday server issue.
>>
>>
>>
>> *mp4-playback-*persistent-usage-record**
>>
>> "persistent-usage-record" sessions are only supported in Edge.
>>
>>
>>
>> Last time the tests were run, none of the three tests were passing on
>> Edge.
>>
>>
>>
>>
>>
>> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <
>> jdsmith@microsoft.com> wrote:
>>
>> I’ve posted updated results for EME tests:
>>
>>
>>
>> -          http://w3c.github.io/test-results/encrypted-media/all.html
>>
>> o   Test files: 70; Total subtests: 220
>>
>> -          http://w3c.github.io/test-results/encrypted-media/less-than-
>> 2.html
>>
>> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
>> Failure level: 64/220 (29.09%)
>>
>> -          http://w3c.github.io/test-results/encrypted-media/complete-f
>> ails.html
>>
>> o   Completely failed files: 44; Completely failed subtests: 29; Failure
>> level: 29/220 (13.18%)
>>
>>
>>
>> Some comments about these results:
>>
>>
>>
>> 1.       They don’t filter for single valid test outcomes (either pass
>> or fail), and timeouts as failures.  Most of the complete-fails are test
>> timeouts.
>>
>> 2.       Test cases have only been partially (~25%) migrated to full drm
>> from Clear Key versions contributed by Google.  The original Google tests
>> are run if not migrated, and can be distinguished by “Google” in the test
>> file path.
>>
>>
>>
>> Jerry
>>
>>
>>
>>
>>
>
>