Re: EME Test Status Update - 10-27-16

Ignoring persistent tests, I see the following tests still failing in an up
to date Firefox Nighty:

/encrypted-media/clearkey-mp4-unique-origin.html    FAIL        0/1
/encrypted-media/drm-events.html    TIMEOUT        0/1
/encrypted-media/drm-mp4-unique-origin.html    FAIL        0/1
/encrypted-media/drm-temporary-license-type.html    FAIL        0/1
/encrypted-media/drm-mp4-playback-temporary-events.html    FAIL        0/1
/encrypted-media/drm-mp4-playback-temporary-expired.html    FAIL        0/1

*-mp4-unique-origin.html is testing Chrome's position on behaviour in the
secure contexts spec which is in dispute. This test should be rewritten to
test behaviour that's not in dispute, or removed. I'd prefer to remove it.
See issue 4136 <https://github.com/w3c/web-platform-tests/issues/4136>.
drm-events.html is caused by the test not completing within 10 seconds due
to network latency. I suggest we increase the timeout for all DRM
tests in issue
4135 <https://github.com/w3c/web-platform-tests/issues/4135>.
drm-temporary-license-type.html - I agree with ddorwin's description above.
drm-mp4-playback-temporary-events.html - I agree with ddorwin's description
above.
drm-mp4-playback-temporary-expired.html - This is now failing with an
"Reached unreachable code" assertion in Firefox. In the past this sort of
failure has indicated a bug in the test itself. I will look tomorrow.

FYI, drm-mp4-playback-temporary-multikey-sequential-readyState.html is now
passing in Firefox Nightly. We've implemented the readyState behaviour; we
now are dropping back to readyState HAVE_CURRENT_DATA and not dispatching
"waitingforkey" until all decrypted frames have been rendered.


Chris.


On Tue, Nov 1, 2016 at 12:57 PM, David Dorwin <ddorwin@google.com> wrote:

>
>
> On Mon, Oct 31, 2016 at 4:24 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
>> We seem very close to finishing.  Do we have a shot at closing open test
>> issues this week?
>>
>>
>>
>> David;  Can you clarify if you consider the new issues you’ve listed as
>> V1?  I think you’ve added these two to the list:
>>
>>
>>
>> -          drm-mp4-playback-temporary-expired.html
>>
>> o   Flaky due to timing.
>>
> This may be a limitation of what we can do in a short test that depends on
> an external server. Someone could take a closer look, but it may just be
> that we need to rerun this if we see failures on Chrome or Firefox.
>
>> -          drm-mp4-unique-origin.html
>>
>> o   New failure
>>
> Chris fixed this test, which was broken by a testharness update, but there
> is now a question of whether we should disable/delete it.
>
>>
>>
>> My previous list:
>>
>>
>>
>> -          drm-mp4-playback-persistent-license-events.html
>>
>> o   https://crbug.com/660393
>>
>> o   ETA?
>>
> This will require pushing a new CDM. We should consider this an
> implementation bug with commitment to fix.
>
>> -          drm-temporary-license-type.html
>>
>>                             Widevine CDM returns a temporary license when
>> a persistent one is requested.
>>
>>                             Proposal in https://github.com/w3c/web-pla
>> tform-tests/issues/4027 to rework the test.
>>
>>                             Is this must fix for V1?
>>
>
> Unless someone volunteers to implement Mark's proposal, we should probably
> disable it for now.
>
>> -          drm-mp4-playback-temporary-events.html
>>
>> o   https://crbug.com/659828
>>
>> o   ETA?
>>
> This will require pushing a new CDM. We should consider this an
> implementation bug with commitment to fix. Chrome and Firefox, I believe,
> have already fixed Clear Key.
>
>> -          drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>
> Chris was working on readyState in Firefox, so hopefully he can give an
> update.
>
> Chrome actually implements readyState but fails this test due to
> https://crbug.com/658422, which is caused by a non-EME implementation
> decision. We intend to fix the EME side but do not have an ETA.
> https://crbug.com/658422#c2 is a patch that demonstrates this difference.
>
>> -          drm-mp4-playback-temporary-multikey-sequential.html
>>
>> o   Am I correct that multiple PRs are targeted at this issue?
>>
>> o   Chris Pearce reports fixes are in work.
>>
>> o   What’s the prospects for closing this week?
>>
> The spec and test were changed to expect HAVE_CURRENT_DATA. Chris, what is
> the status of Firefox?
>
> Chrome fails for the same reason as the previous test.
>
>>
>>
>> I added a V1 milestone to the test repository and marked one issue V1 (
>> https://github.com/w3c/web-platform-tests/issues/4027) as V1.  There are
>> 10 other test issues without a milestone, and 5 marked VNext.  Can folks
>> weigh in on which should be treated as V1?
>>
>
> I punted a bunch of request for new tests.  The remaining four issues (
> https://github.com/w3c/web-platform-tests/issues?q=is%
> 3Aopen+label%3Aencrypted-media+no%3Amilestone includes PRs) are minor
> issues we could try to fix but aren't required, have a PR out for, or might
> have a test for, so I didn't punt them.
>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* David Dorwin [mailto:ddorwin@google.com]
>> *Sent:* Friday, October 28, 2016 9:10 AM
>> *To:* Chris Pearce <cpearce@mozilla.com>
>> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <
>> Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt
>> Wolenetz <wolenetz@google.com>; public-html-media@w3.org
>> *Subject:* Re: EME Test Status Update - 10-27-16
>>
>>
>>
>> Thanks, Jerry.
>>
>>
>>
>> The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html
>> failure is tracked in https://crbug.com/659828.
>>
>>
>>
>> The drm-mp4-playback-persistent-license-events.html failure on Chrome OS
>> is tracked as https://crbug.com/660393.
>>
>>
>>
>> drm-mp4-playback-temporary-expired.html is flaky due to timing. I just
>> saw a failure and a pass.
>>
>>
>>
>> drm-mp4-unique-origin.html appears to be a new failure. We're looking
>> into it.
>>
>>
>>
>> On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com>
>> wrote:
>>
>> The failure in /encrypted-media/drm-mp4-playback-temporary-events.html
>> (at least in Chrome and in Firefox) is caused by Widevine CDM resolving
>> MediaKeySession.closed and the promise returned by MediaKeySession.close()
>> in the wrong order.
>>
>> The test changed with issue 4058
>> <https://github.com/w3c/web-platform-tests/issues/4058> to expect that
>> the MediaKeySession.closed promise is resolved before the promise returned
>> by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on
>> the Widevine CDM to resolve the promise in the correct order, and it
>> doesn't. So this is an implementation bug in the Widevine CDM.
>>
>> The ClearKey variant of this test
>> <https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
>> should be fixed in Firefox Nightly in a day or so, and is already fixed in
>> the latest Chrome Canary.
>>
>>
>>
>> We have patches in the works for fixing drm-mp4-playback-temporary-mul
>> tikey-sequential[-readyState].html.
>>
>>
>>
>> Chris.
>>
>>
>>
>>
>>
>> On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
>> wrote:
>>
>> Summary of issues related to each failure is:
>>
>>
>>
>> Persistent-license failures - 1 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>>
>>
>>      This is being internally investigated at Google for a Chrome OS fix.
>>
>>
>>
>> Temporary session failures - 2 Failures
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>>
>>
>>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
>> assigned to mwatson2
>>
>>
>>
>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, temporary, mp4, playback, check events
>>
>>
>>
>>      This is actually a new failure that didn’t occur on the 10/24/2016
>> pass.  Can anyone comment on the test?
>>
>>
>>
>> Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>>
>>
>>      This issue is dependent on some spec clarifications in these issues:
>>
>>      https://github.com/w3c/encrypted-media/issues/338 - closed
>>
>>      https://github.com/w3c/encrypted-media/issues/339 - closed
>>
>>      https://github.com/w3c/encrypted-media/issues/336 - in review
>>
>>
>>
>>      I’m not sure if we have an issue opened to modify the test once the
>> spec issues are cleared.
>>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
>> *Sent:* Thursday, October 27, 2016 4:40 PM
>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>> wolenetz@google.com>; 'public-html-media@w3.org' <
>> public-html-media@w3.org>
>> *Subject:* EME Test Status Update - 10-27-16
>>
>>
>>
>> I’ve posted updated test status reports to reflect a few bug fixes and
>> also to add Chrome OS results.
>>
>>
>>
>> Reports:
>>
>>
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>
>> o   *Test files: 97; Total subtests: 306*
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/complete-
>> fails.html
>>
>> o   *Completely failed files: 16; Completely failed subtests: 16;
>> Failure level: 16/306 (5.23%)*
>>
>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>> 2.html
>>
>> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
>> Failure level: 40/306 (13.07%)*
>>
>> Comments:
>>
>>
>>
>> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from
>> 47 to 40 subtest failures.
>>
>> -   Persistent-license failures dropped from 5 to 1 with the addition of
>> Chrome on Chrome OS results.
>>
>> -   One temporary session test failed that was previously passing:
>>
>> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> o   drm, temporary, mp4, playback, check events
>>
>> -   There are only 5 tests in complete-fails that we are still working
>> on for our conformance record.
>>
>> -   That reflects our revised criteria of having a single passing
>> implementation (in addition to broad interoperable experience).
>>
>> In line with that criteria, I’ve chosen to break down just the
>> complete-fails test results for this summary.
>>
>>
>>
>> Browser Versions:
>>
>>
>>
>> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>>
>> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>>
>> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>>
>> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>>
>>
>>
>> Summary Groupings;
>>
>>
>>
>> Errors (considered bugs, not blocking)
>> 0
>>
>> Clear Key persistent-license (not planned, may make
>> non-normative)         5
>>
>> Clear Key persistent-usage-record (at risk, planned to be
>> removed)         3
>>
>> Clear Key transitions between clear and encrypted (bugs, not
>> blocking)     0
>>
>>
>>
>> Not Blocking Tests
>>  8
>>
>>
>>
>> Persistent-license (fixes in work)
>>           1
>>
>> Persistent-usage-record (at risk, planned to be
>> removed)                   3
>>
>> Temporary session edge cases (bugs, not block, fixes in
>> work)              2
>>
>> Temporary session readyState (issues under discussion, at risk)
>>  2
>>
>>
>>
>> In work or at risk
>> 8
>>
>>
>>
>> TOTAL
>> 16
>>
>>
>>
>>
>>
>> Details on failed tests are below.
>>
>>
>>
>> Jerry
>>
>>
>>
>>
>>
>> Error handling (considered bugs, not blocking) - 0 Failures
>>
>>
>>
>> Clear key persistent-license tests (not planned to be implemented, may
>> make informative note) - 5 Failures
>>
>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>> acknowledge
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>> and destroy
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>>
>>
>>
>> Clear key persistent-usage-record (not planned to be implemented) - 3
>> Failures
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4playback
>>
>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
>> window
>>
>>
>>
>> Clear key transitions between clear and encrypted content (considered
>> bugs, not blocking) - 2 Failures
>>
>>
>>
>> Persistent-license failures - 6 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-license, mp4, playback, check events
>>
>>
>>
>> Persistent-usage-record failures - 3 Failures
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, check events
>>
>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>
>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>> 100.00%, 0.33% of total)
>>
>> drm, persistent-usage-record, mp4playback
>>
>>
>>
>> Temporary session failures - 2 Failures
>>
>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33% of
>> total)
>>
>> drm, cannot load persistent license into temporary session
>>
>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>> 0.33% of total)
>>
>> drm, temporary, mp4, playback, check events
>>
>>
>>
>> Temporary session readyState failures (listed at risk, under discussion)
>> - 2 Failures
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>> readyState
>>
>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>> (1/1, 100.00%, 0.33% of total)
>>
>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>
>>
>>
>>
>>
>
>