Re: EME Test Status Update - 10-27-16

As best I can tell, drm-mp4-playback-temporary-expired.html is failing
intermittently because the DRMToday server sometimes returns a 403 response
on the POST request to retrieve the license. If I retry, I get a 200
response and the test works. Not sure what's going on there, but I suspect
it's an issue with the server, as I can't see a difference between the
requests that fail and the requests that succeed on the client side.
Typically its only the first request that fails.

On Tue, Nov 1, 2016 at 9:39 PM, Chris Pearce <cpearce@mozilla.com> wrote:

> Ignoring persistent tests, I see the following tests still failing in an
> up to date Firefox Nighty:
>
> /encrypted-media/clearkey-mp4-unique-origin.html    FAIL        0/1
> /encrypted-media/drm-events.html    TIMEOUT        0/1
> /encrypted-media/drm-mp4-unique-origin.html    FAIL        0/1
> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
> /encrypted-media/drm-mp4-playback-temporary-events.html    FAIL        0/1
> /encrypted-media/drm-mp4-playback-temporary-expired.html    FAIL
> 0/1
>
> *-mp4-unique-origin.html is testing Chrome's position on behaviour in the
> secure contexts spec which is in dispute. This test should be rewritten to
> test behaviour that's not in dispute, or removed. I'd prefer to remove it.
> See issue 4136 <https://github.com/w3c/web-platform-tests/issues/4136>.
> drm-events.html is caused by the test not completing within 10 seconds due
> to network latency. I suggest we increase the timeout for all DRM tests in issue
> 4135 <https://github.com/w3c/web-platform-tests/issues/4135>.
> drm-temporary-license-type.html - I agree with ddorwin's description
> above.
> drm-mp4-playback-temporary-events.html - I agree with ddorwin's
> description above.
> drm-mp4-playback-temporary-expired.html - This is now failing with an
> "Reached unreachable code" assertion in Firefox. In the past this sort of
> failure has indicated a bug in the test itself. I will look tomorrow.
>
> FYI, drm-mp4-playback-temporary-multikey-sequential-readyState.html is
> now passing in Firefox Nightly. We've implemented the readyState behaviour;
> we now are dropping back to readyState HAVE_CURRENT_DATA and not
> dispatching "waitingforkey" until all decrypted frames have been rendered.
>
>
> Chris.
>
>
> On Tue, Nov 1, 2016 at 12:57 PM, David Dorwin <ddorwin@google.com> wrote:
>
>>
>>
>> On Mon, Oct 31, 2016 at 4:24 PM, Jerry Smith (WPT) <jdsmith@microsoft.com
>> > wrote:
>>
>>> We seem very close to finishing.  Do we have a shot at closing open test
>>> issues this week?
>>>
>>>
>>>
>>> David;  Can you clarify if you consider the new issues you’ve listed as
>>> V1?  I think you’ve added these two to the list:
>>>
>>>
>>>
>>> -          drm-mp4-playback-temporary-expired.html
>>>
>>> o   Flaky due to timing.
>>>
>> This may be a limitation of what we can do in a short test that depends
>> on an external server. Someone could take a closer look, but it may just be
>> that we need to rerun this if we see failures on Chrome or Firefox.
>>
>>> -          drm-mp4-unique-origin.html
>>>
>>> o   New failure
>>>
>> Chris fixed this test, which was broken by a testharness update, but
>> there is now a question of whether we should disable/delete it.
>>
>>>
>>>
>>> My previous list:
>>>
>>>
>>>
>>> -          drm-mp4-playback-persistent-license-events.html
>>>
>>> o   https://crbug.com/660393
>>>
>>> o   ETA?
>>>
>> This will require pushing a new CDM. We should consider this an
>> implementation bug with commitment to fix.
>>
>>> -          drm-temporary-license-type.html
>>>
>>>                             Widevine CDM returns a temporary license
>>> when a persistent one is requested.
>>>
>>>                             Proposal in https://github.com/w3c/web-pla
>>> tform-tests/issues/4027 to rework the test.
>>>
>>>                             Is this must fix for V1?
>>>
>>
>> Unless someone volunteers to implement Mark's proposal, we should
>> probably disable it for now.
>>
>>> -          drm-mp4-playback-temporary-events.html
>>>
>>> o   https://crbug.com/659828
>>>
>>> o   ETA?
>>>
>> This will require pushing a new CDM. We should consider this an
>> implementation bug with commitment to fix. Chrome and Firefox, I believe,
>> have already fixed Clear Key.
>>
>>> -          drm-mp4-playback-temporary-multikey-sequential-readyState.ht
>>> ml
>>>
>> Chris was working on readyState in Firefox, so hopefully he can give an
>> update.
>>
>> Chrome actually implements readyState but fails this test due to
>> https://crbug.com/658422, which is caused by a non-EME implementation
>> decision. We intend to fix the EME side but do not have an ETA.
>> https://crbug.com/658422#c2 is a patch that demonstrates this difference.
>>
>>> -          drm-mp4-playback-temporary-multikey-sequential.html
>>>
>>> o   Am I correct that multiple PRs are targeted at this issue?
>>>
>>> o   Chris Pearce reports fixes are in work.
>>>
>>> o   What’s the prospects for closing this week?
>>>
>> The spec and test were changed to expect HAVE_CURRENT_DATA. Chris, what
>> is the status of Firefox?
>>
>> Chrome fails for the same reason as the previous test.
>>
>>>
>>>
>>> I added a V1 milestone to the test repository and marked one issue V1 (
>>> https://github.com/w3c/web-platform-tests/issues/4027) as V1.  There
>>> are 10 other test issues without a milestone, and 5 marked VNext.  Can
>>> folks weigh in on which should be treated as V1?
>>>
>>
>> I punted a bunch of request for new tests.  The remaining four issues (
>> https://github.com/w3c/web-platform-tests/issues?q=is%3Aope
>> n+label%3Aencrypted-media+no%3Amilestone includes PRs) are minor issues
>> we could try to fix but aren't required, have a PR out for, or might have a
>> test for, so I didn't punt them.
>>
>>>
>>>
>>> Jerry
>>>
>>>
>>>
>>> *From:* David Dorwin [mailto:ddorwin@google.com]
>>> *Sent:* Friday, October 28, 2016 9:10 AM
>>> *To:* Chris Pearce <cpearce@mozilla.com>
>>> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <
>>> Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt
>>> Wolenetz <wolenetz@google.com>; public-html-media@w3.org
>>> *Subject:* Re: EME Test Status Update - 10-27-16
>>>
>>>
>>>
>>> Thanks, Jerry.
>>>
>>>
>>>
>>> The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html
>>> failure is tracked in https://crbug.com/659828.
>>>
>>>
>>>
>>> The drm-mp4-playback-persistent-license-events.html failure on Chrome
>>> OS is tracked as https://crbug.com/660393.
>>>
>>>
>>>
>>> drm-mp4-playback-temporary-expired.html is flaky due to timing. I just
>>> saw a failure and a pass.
>>>
>>>
>>>
>>> drm-mp4-unique-origin.html appears to be a new failure. We're looking
>>> into it.
>>>
>>>
>>>
>>> On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com>
>>> wrote:
>>>
>>> The failure in /encrypted-media/drm-mp4-playback-temporary-events.html
>>> (at least in Chrome and in Firefox) is caused by Widevine CDM resolving
>>> MediaKeySession.closed and the promise returned by MediaKeySession.close()
>>> in the wrong order.
>>>
>>> The test changed with issue 4058
>>> <https://github.com/w3c/web-platform-tests/issues/4058> to expect that
>>> the MediaKeySession.closed promise is resolved before the promise returned
>>> by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on
>>> the Widevine CDM to resolve the promise in the correct order, and it
>>> doesn't. So this is an implementation bug in the Widevine CDM.
>>>
>>> The ClearKey variant of this test
>>> <https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
>>> should be fixed in Firefox Nightly in a day or so, and is already fixed in
>>> the latest Chrome Canary.
>>>
>>>
>>>
>>> We have patches in the works for fixing drm-mp4-playback-temporary-mul
>>> tikey-sequential[-readyState].html.
>>>
>>>
>>>
>>> Chris.
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <
>>> jdsmith@microsoft.com> wrote:
>>>
>>> Summary of issues related to each failure is:
>>>
>>>
>>>
>>> Persistent-license failures - 1 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, check events
>>>
>>>
>>>
>>>      This is being internally investigated at Google for a Chrome OS fix.
>>>
>>>
>>>
>>> Temporary session failures - 2 Failures
>>>
>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>> of total)
>>>
>>> drm, cannot load persistent license into temporary session
>>>
>>>
>>>
>>>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
>>> assigned to mwatson2
>>>
>>>
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>>> 0.33% of total)
>>>
>>> drm, temporary, mp4, playback, check events
>>>
>>>
>>>
>>>      This is actually a new failure that didn’t occur on the 10/24/2016
>>> pass.  Can anyone comment on the test?
>>>
>>>
>>>
>>> Temporary session readyState failures (listed at risk, under discussion)
>>> - 2 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>> readyState
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>
>>>
>>>
>>>      This issue is dependent on some spec clarifications in these issues:
>>>
>>>      https://github.com/w3c/encrypted-media/issues/338 - closed
>>>
>>>      https://github.com/w3c/encrypted-media/issues/339 - closed
>>>
>>>      https://github.com/w3c/encrypted-media/issues/336 - in review
>>>
>>>
>>>
>>>      I’m not sure if we have an issue opened to modify the test once the
>>> spec issues are cleared.
>>>
>>>
>>>
>>> Jerry
>>>
>>>
>>>
>>> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
>>> *Sent:* Thursday, October 27, 2016 4:40 PM
>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>>> wolenetz@google.com>; 'public-html-media@w3.org' <
>>> public-html-media@w3.org>
>>> *Subject:* EME Test Status Update - 10-27-16
>>>
>>>
>>>
>>> I’ve posted updated test status reports to reflect a few bug fixes and
>>> also to add Chrome OS results.
>>>
>>>
>>>
>>> Reports:
>>>
>>>
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>>
>>> o   *Test files: 97; Total subtests: 306*
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/complete-f
>>> ails.html
>>>
>>> o   *Completely failed files: 16; Completely failed subtests: 16;
>>> Failure level: 16/306 (5.23%)*
>>>
>>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>>> 2.html
>>>
>>> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
>>> Failure level: 40/306 (13.07%)*
>>>
>>> Comments:
>>>
>>>
>>>
>>> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from
>>> 47 to 40 subtest failures.
>>>
>>> -   Persistent-license failures dropped from 5 to 1 with the addition
>>> of Chrome on Chrome OS results.
>>>
>>> -   One temporary session test failed that was previously passing:
>>>
>>> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> o   drm, temporary, mp4, playback, check events
>>>
>>> -   There are only 5 tests in complete-fails that we are still working
>>> on for our conformance record.
>>>
>>> -   That reflects our revised criteria of having a single passing
>>> implementation (in addition to broad interoperable experience).
>>>
>>> In line with that criteria, I’ve chosen to break down just the
>>> complete-fails test results for this summary.
>>>
>>>
>>>
>>> Browser Versions:
>>>
>>>
>>>
>>> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>>>
>>> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>>>
>>> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>>>
>>> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>>>
>>>
>>>
>>> Summary Groupings;
>>>
>>>
>>>
>>> Errors (considered bugs, not blocking)
>>> 0
>>>
>>> Clear Key persistent-license (not planned, may make
>>> non-normative)         5
>>>
>>> Clear Key persistent-usage-record (at risk, planned to be
>>> removed)         3
>>>
>>> Clear Key transitions between clear and encrypted (bugs, not
>>> blocking)     0
>>>
>>>
>>>
>>> Not Blocking Tests
>>>  8
>>>
>>>
>>>
>>> Persistent-license (fixes in work)
>>>           1
>>>
>>> Persistent-usage-record (at risk, planned to be
>>> removed)                   3
>>>
>>> Temporary session edge cases (bugs, not block, fixes in
>>> work)              2
>>>
>>> Temporary session readyState (issues under discussion, at
>>> risk)            2
>>>
>>>
>>>
>>> In work or at risk
>>> 8
>>>
>>>
>>>
>>> TOTAL
>>> 16
>>>
>>>
>>>
>>>
>>>
>>> Details on failed tests are below.
>>>
>>>
>>>
>>> Jerry
>>>
>>>
>>>
>>>
>>>
>>> Error handling (considered bugs, not blocking) - 0 Failures
>>>
>>>
>>>
>>> Clear key persistent-license tests (not planned to be implemented, may
>>> make informative note) - 5 Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>>> acknowledge
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4playback
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>>> and destroy
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and playback
>>>
>>>
>>>
>>> Clear key persistent-usage-record (not planned to be implemented) - 3
>>> Failures
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>>
>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4playback
>>>
>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in new
>>> window
>>>
>>>
>>>
>>> Clear key transitions between clear and encrypted content (considered
>>> bugs, not blocking) - 2 Failures
>>>
>>>
>>>
>>> Persistent-license failures - 6 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-license, mp4, playback, check events
>>>
>>>
>>>
>>> Persistent-usage-record failures - 3 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4, playback, check events
>>>
>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>>
>>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>>> 100.00%, 0.33% of total)
>>>
>>> drm, persistent-usage-record, mp4playback
>>>
>>>
>>>
>>> Temporary session failures - 2 Failures
>>>
>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>> of total)
>>>
>>> drm, cannot load persistent license into temporary session
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>>> 0.33% of total)
>>>
>>> drm, temporary, mp4, playback, check events
>>>
>>>
>>>
>>> Temporary session readyState failures (listed at risk, under discussion)
>>> - 2 Failures
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>> readyState
>>>
>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>> (1/1, 100.00%, 0.33% of total)
>>>
>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>
>>>
>>>
>>>
>>>
>>
>>
>