Re: [EME] Addressing Less Than 2 Passes tests

Hi,

Apologies for my tardy response, I've been on leave.

On Wed, Sep 14, 2016 at 4:55 PM, David Dorwin <ddorwin@google.com> wrote:

> [WAS: Updated EME Test Status]
>
> The following is an analysis of the less-than-2 results
> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry
> published earlier today. There are specific requests for *Jerry* and
> *Chris* below, but we could use help from everyone.
>
> If you can help with the DRMtoday server issues or migrate the tests to
> use the new test account
> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us
> know ASAP as this is blocking all drm-* tests.
>
> Until the DRMtoday test server issue is resolved, we'll have to mostly
> ignore the drm-* tests since we do not have good results. However, I've
> commented on some that were failing in the test results before the today's
> update.
>
> In addition to the failures below, we may also see additional test
> failures as the Google/ tests are migrated and start running on Edge and
> Firefox as well exercising commercial Key Systems.
>
> *Google/**
> All of the test in Google/ are expected to potentially fail on other
> browsers until the tests are migrated. We need help migrating the remaining
> tests: https://github.com/w3c/web-platform-tests/issues/3583
> #issuecomment-243577488.
>
> (I merged an update to Google/encrypted-media-syntax.html that fixes the
> new failure reported in Chrome 55.)
>
> *idlharness.html*
>
>    - Three "interface: attribute" tests for the new event handler
>    attributes:
>       - Chrome passes these.
>       - We need another browser to implement the three new event handler
>       attributes. This is trivial, so while it should not block PR, it should be
>       easy to fix.
>       - *Jerry* and *Chris*, do you have plans to implement these?
>
>
Yes, it should be easy for us to fix.


>
>    - Nine "interface: operation" tests:
>       - Firefox passes these.
>       - The failures in Chrome are known
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and
>       caused by a Blink issue
>       <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated
>       to EME.
>       - Edge also fails these tests. *Jerry*, do you know why?
>       - I think we can just note this in the test report.
>
> (While not an issue for PR since Edge and Firefox pass, for future
> reference, the six "existence and properties of interface prototype object"
> test failures in Chrome are also known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and
> unrelated to EME.)
>
> *clearkey-*persistent**
> These eight tests (all complete-fails) are all related to persistent-*
> session types. No Clear Key implementation supports anything other than
> temporary sessions, which makes sense, and I don't expect this to change.
> Thus, these tests will not pass in the v1 timeframe.
>
> *clearkey-keystatuses.html*
> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox*
> to pass this test. It currently fails with:
>
>> assert_equals: keystatus value for invalid key should be undefined (1)
>> expected (undefined) undefined but got (string) "internal-error"
>>
>
> *Chris*, please take a look.
>


This now passes in Firefox 51 (in our Developer Edition/Aurora channel
tomorrow I believe).


>
> *drm-keystatuses.html*
> Mark wrote
> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790>
> :
>
>> ... it appears that all three browsers are non-compliant:
>>
>>    - Chrome does not generate a keystatuseschange event after close() is
>>    called (for the DRM case, it does for the ClearKey case)
>>
>>
>>    - Firefox and Edge both have incorrect values in the keystatuses map
>>    (in different ways).
>>
>>
> The Chrome issue is likely the same as this known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue.
> *Jerry* and *Chris*, please take a look.
>

This is caused by Firefox dispatching one keystatuseschange event for every
key that changes status, rather than aggregating keys changing status at
the same time into a single keystatsueschange event. The spec says we
should be aggregating the status changes into a single event, so we'll
update to match the spec. We're working on this in bug 1303922
<https://bugzilla.mozilla.org/show_bug.cgi?id=1303922>.



>
> *drm-mp4-playback-temporary-events.html*
> Last time the tests were run, this was passing on Firefox, failing on
> Chrome, and timing out on Edge.
>
> This Chrome issue is known
> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under
> investigation.
> *Jerry*, do you know why Edge was timing out?
>
> *drm-mp4-playback-temporary-waitingforkey.html*
> Last time the tests were run, this was timing out on all three browsers,
> but Mark landed a fix.
>

Firefox Nightly passes this test.


>
> *drm-mp4-playback-*persistent-license**
> "persistent-license" sessions are only supported on Chrome on Chrome OS,
> and I'm not aware of any plans to support them in other implementations
> before PR.
>
> At least one of these were failing the last step on Chrome OS due to a
> DRMtoday server issue.
>
> *mp4-playback-*persistent-usage-record**
> "persistent-usage-record" sessions are only supported in Edge.
>
> Last time the tests were run, none of the three tests were passing on Edge.
>
>
> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com
> > wrote:
>
>> I’ve posted updated results for EME tests:
>>
>>
>>
>> -          http://w3c.github.io/test-results/encrypted-media/all.html
>>
>> o   Test files: 70; Total subtests: 220
>>
>> -          http://w3c.github.io/test-results/encrypted-media/less-than-
>> 2.html
>>
>> o   Test files without 2 passes: 44; Subtests without 2 passes: 64;
>> Failure level: 64/220 (29.09%)
>>
>> -          http://w3c.github.io/test-results/encrypted-media/complete-f
>> ails.html
>>
>> o   Completely failed files: 44; Completely failed subtests: 29; Failure
>> level: 29/220 (13.18%)
>>
>>
>>
>> Some comments about these results:
>>
>>
>>
>> 1.       They don’t filter for single valid test outcomes (either pass
>> or fail), and timeouts as failures.  Most of the complete-fails are test
>> timeouts.
>>
>> 2.       Test cases have only been partially (~25%) migrated to full drm
>> from Clear Key versions contributed by Google.  The original Google tests
>> are run if not migrated, and can be distinguished by “Google” in the test
>> file path.
>>
>>
>>
>> Jerry
>>
>
>
Chris Pearce.