W3C home > Mailing lists > Public > public-html-media@w3.org > September 2016

Re: [EME] Addressing Less Than 2 Passes tests

From: Chris Pearce <cpearce@mozilla.com>
Date: Fri, 23 Sep 2016 08:36:25 +1200
Message-ID: <CAFWdrwYy7nnF_di26jJRTxyzRPQ=Q36g_E9WrZ2wKCUAUyWaVg@mail.gmail.com>
To: David Dorwin <ddorwin@google.com>
Cc: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "<public-html-media@w3.org>" <public-html-media@w3.org>
On Wed, Sep 21, 2016 at 1:50 PM, Chris Pearce <cpearce@mozilla.com> wrote:

> Hi,
>
> Apologies for my tardy response, I've been on leave.
>
> On Wed, Sep 14, 2016 at 4:55 PM, David Dorwin <ddorwin@google.com> wrote:
>
>> [WAS: Updated EME Test Status]
>>
>> The following is an analysis of the less-than-2 results
>> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
>> published earlier today. There are specific requests for *Jerry* and
>> *Chris* below, but we could use help from everyone.
>>
>> If you can help with the DRMtoday server issues or migrate the tests to
>> use the new test account
>> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us
>> know ASAP as this is blocking all drm-* tests.
>>
>> Until the DRMtoday test server issue is resolved, we'll have to mostly
>> ignore the drm-* tests since we do not have good results. However, I've
>> commented on some that were failing in the test results before the today's
>> update.
>>
>> In addition to the failures below, we may also see additional test
>> failures as the Google/ tests are migrated and start running on Edge and
>> Firefox as well exercising commercial Key Systems.
>>
>> *Google/**
>> All of the test in Google/ are expected to potentially fail on other
>> browsers until the tests are migrated. We need help migrating the remaining
>> tests: https://github.com/w3c/web-platform-tests/issues/3583
>> #issuecomment-243577488.
>>
>> (I merged an update to Google/encrypted-media-syntax.html that fixes the
>> new failure reported in Chrome 55.)
>>
>> *idlharness.html*
>>
>>    - Three "interface: attribute" tests for the new event handler
>>    attributes:
>>       - Chrome passes these.
>>       - We need another browser to implement the three new event handler
>>       attributes. This is trivial, so while it should not block PR, it should be
>>       easy to fix.
>>       - *Jerry* and *Chris*, do you have plans to implement these?
>>
>>
> Yes, it should be easy for us to fix.
>


I have implemented the three new event handler attributes. They should
appear in Firefox Nightly in the next day or so.


Chris.



>
>
>>
>>    - Nine "interface: operation" tests:
>>       - Firefox passes these.
>>       - The failures in Chrome are known
>>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>>       caused by a Blink issue
>>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>>       to EME.
>>       - Edge also fails these tests. *Jerry*, do you know why?
>>       - I think we can just note this in the test report.
>>
>> (While not an issue for PR since Edge and Firefox pass, for future
>> reference, the six "existence and properties of interface prototype object"
>> test failures in Chrome are also known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
>> unrelated to EME.)
>>
>> *clearkey-*persistent**
>> These eight tests (all complete-fails) are all related to persistent-*
>> session types. No Clear Key implementation supports anything other than
>> temporary sessions, which makes sense, and I don't expect this to change.
>> Thus, these tests will not pass in the v1 timeframe.
>>
>> *clearkey-keystatuses.html*
>> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
>> to pass this test. It currently fails with:
>>
>>> assert_equals: keystatus value for invalid key should be undefined (1)
>>> expected (undefined) undefined but got (string) "internal-error"
>>>
>>
>> *Chris*, please take a look.
>>
>
>
> This now passes in Firefox 51 (in our Developer Edition/Aurora channel
> tomorrow I believe).
>
>
>>
>> *drm-keystatuses.html*
>> Mark wrote
>> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
>> :
>>
>>> ... it appears that all three browsers are non-compliant:
>>>
>>>    - Chrome does not generate a keystatuseschange event after close() is
>>>    called (for the DRM case, it does for the ClearKey case)
>>>
>>>
>>>    - Firefox and Edge both have incorrect values in the keystatuses map
>>>    (in different ways).
>>>
>>>
>> The Chrome issue is likely the same as this known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
>> *Jerry* and *Chris*, please take a look.
>>
>
> This is caused by Firefox dispatching one keystatuseschange event for
> every key that changes status, rather than aggregating keys changing status
> at the same time into a single keystatsueschange event. The spec says we
> should be aggregating the status changes into a single event, so we'll
> update to match the spec. We're working on this in bug 1303922
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1303922>.
>
>
>
>>
>> *drm-mp4-playback-temporary-events.html*
>> Last time the tests were run, this was passing on Firefox, failing on
>> Chrome, and timing out on Edge.
>>
>> This Chrome issue is known
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
>> investigation.
>> *Jerry*, do you know why Edge was timing out?
>>
>> *drm-mp4-playback-temporary-waitingforkey.html*
>> Last time the tests were run, this was timing out on all three browsers,
>> but Mark landed a fix.
>>
>
> Firefox Nightly passes this test.
>
>
>>
>> *drm-mp4-playback-*persistent-license**
>> "persistent-license" sessions are only supported on Chrome on Chrome OS,
>> and I'm not aware of any plans to support them in other implementations
>> before PR.
>>
>> At least one of these were failing the last step on Chrome OS due to a
>> DRMtoday server issue.
>>
>> *mp4-playback-*persistent-usage-record**
>> "persistent-usage-record" sessions are only supported in Edge.
>>
>> Last time the tests were run, none of the three tests were passing on
>> Edge.
>>
>>
>> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <
>> jdsmith@microsoft.com> wrote:
>>
>>> I’ve posted updated results for EME tests:
>>>
>>>
>>>
>>> -          http://w3c.github.io/test-results/encrypted-media/all.html
>>>
>>> o   Test files: 70; Total subtests: 220
>>>
>>> -          http://w3c.github.io/test-results/encrypted-media/less-than-
>>> 2.html
>>>
>>> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
>>> Failure level: 64/220 (29.09%)
>>>
>>> -          http://w3c.github.io/test-results/encrypted-media/complete-f
>>> ails.html
>>>
>>> o   Completely failed files: 44; Completely failed subtests: 29;
>>> Failure level: 29/220 (13.18%)
>>>
>>>
>>>
>>> Some comments about these results:
>>>
>>>
>>>
>>> 1.       They don’t filter for single valid test outcomes (either pass
>>> or fail), and timeouts as failures.  Most of the complete-fails are test
>>> timeouts.
>>>
>>> 2.       Test cases have only been partially (~25%) migrated to full
>>> drm from Clear Key versions contributed by Google.  The original Google
>>> tests are run if not migrated, and can be distinguished by “Google” in the
>>> test file path.
>>>
>>>
>>>
>>> Jerry
>>>
>>
>>
> Chris Pearce.
>
>
Received on Thursday, 22 September 2016 20:36:55 UTC

This archive was generated by hypermail 2.3.1 : Thursday, 22 September 2016 20:36:56 UTC