Re: EME Test Status Update - 10-27-16

If you ever need me to check server failure reasons, ping me and I can
check in the sever log.

It does fail expiration times in the past, which makes this test difficult
to make reliable. Perhaps it should just be made much longer: request an
expiration 20seconds in the future then wait for it to expire ?

...Mark

Sent from my iPhone

On Nov 1, 2016, at 2:51 PM, John Rummell <jrummell@google.com> wrote:

When I was testing with the DRMToday server, I tried a bunch of expiration
times. Based on the responses I got I think it returns a 403 if the
expiration time is in the past, so anything less than ~1 second (or
negative) consistently failed for me. I wonder if the first time it just
takes a little bit longer to get there, so to the server the expiration
time looks like it's in the past?

On Tue, Nov 1, 2016 at 2:43 PM, Chris Pearce <cpearce@mozilla.com> wrote:

> As best I can tell, drm-mp4-playback-temporary-expired.html is failing
> intermittently because the DRMToday server sometimes returns a 403 response
> on the POST request to retrieve the license. If I retry, I get a 200
> response and the test works. Not sure what's going on there, but I suspect
> it's an issue with the server, as I can't see a difference between the
> requests that fail and the requests that succeed on the client side.
> Typically its only the first request that fails.
>
> On Tue, Nov 1, 2016 at 9:39 PM, Chris Pearce <cpearce@mozilla.com> wrote:
>
>> Ignoring persistent tests, I see the following tests still failing in an
>> up to date Firefox Nighty:
>>
>> /encrypted-media/clearkey-mp4-unique-origin.html    FAIL        0/1
>> /encrypted-media/drm-events.html    TIMEOUT        0/1
>> /encrypted-media/drm-mp4-unique-origin.html    FAIL        0/1
>> /encrypted-media/drm-temporary-license-type.html    FAIL        0/1
>> /encrypted-media/drm-mp4-playback-temporary-events.html    FAIL
>> 0/1
>> /encrypted-media/drm-mp4-playback-temporary-expired.html    FAIL
>> 0/1
>>
>> *-mp4-unique-origin.html is testing Chrome's position on behaviour in the
>> secure contexts spec which is in dispute. This test should be rewritten to
>> test behaviour that's not in dispute, or removed. I'd prefer to remove it.
>> See issue 4136 <https://github.com/w3c/web-platform-tests/issues/4136>.
>> drm-events.html is caused by the test not completing within 10 seconds
>> due to network latency. I suggest we increase the timeout for all DRM tests
>> in issue 4135 <https://github.com/w3c/web-platform-tests/issues/4135>.
>> drm-temporary-license-type.html - I agree with ddorwin's description
>> above.
>> drm-mp4-playback-temporary-events.html - I agree with ddorwin's
>> description above.
>> drm-mp4-playback-temporary-expired.html - This is now failing with an
>> "Reached unreachable code" assertion in Firefox. In the past this sort of
>> failure has indicated a bug in the test itself. I will look tomorrow.
>>
>> FYI, drm-mp4-playback-temporary-multikey-sequential-readyState.html is
>> now passing in Firefox Nightly. We've implemented the readyState behaviour;
>> we now are dropping back to readyState HAVE_CURRENT_DATA and not
>> dispatching "waitingforkey" until all decrypted frames have been rendered.
>>
>>
>> Chris.
>>
>>
>> On Tue, Nov 1, 2016 at 12:57 PM, David Dorwin <ddorwin@google.com> wrote:
>>
>>>
>>>
>>> On Mon, Oct 31, 2016 at 4:24 PM, Jerry Smith (WPT) <
>>> jdsmith@microsoft.com> wrote:
>>>
>>>> We seem very close to finishing.  Do we have a shot at closing open
>>>> test issues this week?
>>>>
>>>>
>>>>
>>>> David;  Can you clarify if you consider the new issues you’ve listed as
>>>> V1?  I think you’ve added these two to the list:
>>>>
>>>>
>>>>
>>>> -          drm-mp4-playback-temporary-expired.html
>>>>
>>>> o   Flaky due to timing.
>>>>
>>> This may be a limitation of what we can do in a short test that depends
>>> on an external server. Someone could take a closer look, but it may just be
>>> that we need to rerun this if we see failures on Chrome or Firefox.
>>>
>>>> -          drm-mp4-unique-origin.html
>>>>
>>>> o   New failure
>>>>
>>> Chris fixed this test, which was broken by a testharness update, but
>>> there is now a question of whether we should disable/delete it.
>>>
>>>>
>>>>
>>>> My previous list:
>>>>
>>>>
>>>>
>>>> -          drm-mp4-playback-persistent-license-events.html
>>>>
>>>> o   https://crbug.com/660393
>>>>
>>>> o   ETA?
>>>>
>>> This will require pushing a new CDM. We should consider this an
>>> implementation bug with commitment to fix.
>>>
>>>> -          drm-temporary-license-type.html
>>>>
>>>>                             Widevine CDM returns a temporary license
>>>> when a persistent one is requested.
>>>>
>>>>                             Proposal in https://github.com/w3c/web-pla
>>>> tform-tests/issues/4027 to rework the test.
>>>>
>>>>                             Is this must fix for V1?
>>>>
>>>
>>> Unless someone volunteers to implement Mark's proposal, we should
>>> probably disable it for now.
>>>
>>>> -          drm-mp4-playback-temporary-events.html
>>>>
>>>> o   https://crbug.com/659828
>>>>
>>>> o   ETA?
>>>>
>>> This will require pushing a new CDM. We should consider this an
>>> implementation bug with commitment to fix. Chrome and Firefox, I believe,
>>> have already fixed Clear Key.
>>>
>>>> -          drm-mp4-playback-temporary-multikey-sequential-readyState.ht
>>>> ml
>>>>
>>> Chris was working on readyState in Firefox, so hopefully he can give an
>>> update.
>>>
>>> Chrome actually implements readyState but fails this test due to
>>> https://crbug.com/658422, which is caused by a non-EME implementation
>>> decision. We intend to fix the EME side but do not have an ETA.
>>> https://crbug.com/658422#c2 is a patch that demonstrates this
>>> difference.
>>>
>>>> -          drm-mp4-playback-temporary-multikey-sequential.html
>>>>
>>>> o   Am I correct that multiple PRs are targeted at this issue?
>>>>
>>>> o   Chris Pearce reports fixes are in work.
>>>>
>>>> o   What’s the prospects for closing this week?
>>>>
>>> The spec and test were changed to expect HAVE_CURRENT_DATA. Chris, what
>>> is the status of Firefox?
>>>
>>> Chrome fails for the same reason as the previous test.
>>>
>>>>
>>>>
>>>> I added a V1 milestone to the test repository and marked one issue V1 (
>>>> https://github.com/w3c/web-platform-tests/issues/4027) as V1.  There
>>>> are 10 other test issues without a milestone, and 5 marked VNext.  Can
>>>> folks weigh in on which should be treated as V1?
>>>>
>>>
>>> I punted a bunch of request for new tests.  The remaining four issues (
>>> https://github.com/w3c/web-platform-tests/issues?q=is%3Aope
>>> n+label%3Aencrypted-media+no%3Amilestone includes PRs) are minor issues
>>> we could try to fix but aren't required, have a PR out for, or might have a
>>> test for, so I didn't punt them.
>>>
>>>>
>>>>
>>>> Jerry
>>>>
>>>>
>>>>
>>>> *From:* David Dorwin [mailto:ddorwin@google.com]
>>>> *Sent:* Friday, October 28, 2016 9:10 AM
>>>> *To:* Chris Pearce <cpearce@mozilla.com>
>>>> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Paul Cotton <
>>>> Paul.Cotton@microsoft.com>; Mark Watson <watsonm@netflix.com>; Matt
>>>> Wolenetz <wolenetz@google.com>; public-html-media@w3.org
>>>> *Subject:* Re: EME Test Status Update - 10-27-16
>>>>
>>>>
>>>>
>>>> Thanks, Jerry.
>>>>
>>>>
>>>>
>>>> The CDM change Chris mentioned for the drm-mp4-playback-temporary-events.html
>>>> failure is tracked in https://crbug.com/659828.
>>>>
>>>>
>>>>
>>>> The drm-mp4-playback-persistent-license-events.html failure on Chrome
>>>> OS is tracked as https://crbug.com/660393.
>>>>
>>>>
>>>>
>>>> drm-mp4-playback-temporary-expired.html is flaky due to timing. I just
>>>> saw a failure and a pass.
>>>>
>>>>
>>>>
>>>> drm-mp4-unique-origin.html appears to be a new failure. We're looking
>>>> into it.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 27, 2016 at 6:43 PM, Chris Pearce <cpearce@mozilla.com>
>>>> wrote:
>>>>
>>>> The failure in /encrypted-media/drm-mp4-playback-temporary-events.html
>>>> (at least in Chrome and in Firefox) is caused by Widevine CDM resolving
>>>> MediaKeySession.closed and the promise returned by MediaKeySession.close()
>>>> in the wrong order.
>>>>
>>>> The test changed with issue 4058
>>>> <https://github.com/w3c/web-platform-tests/issues/4058> to expect that
>>>> the MediaKeySession.closed promise is resolved before the promise returned
>>>> by MediaKeySession.close() is resolved. Both Firefox and Chrome depend on
>>>> the Widevine CDM to resolve the promise in the correct order, and it
>>>> doesn't. So this is an implementation bug in the Widevine CDM.
>>>>
>>>> The ClearKey variant of this test
>>>> <https://w3c-test.org/encrypted-media/clearkey-mp4-playback-temporary-events.html>
>>>> should be fixed in Firefox Nightly in a day or so, and is already fixed in
>>>> the latest Chrome Canary.
>>>>
>>>>
>>>>
>>>> We have patches in the works for fixing drm-mp4-playback-temporary-mul
>>>> tikey-sequential[-readyState].html.
>>>>
>>>>
>>>>
>>>> Chris.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 28, 2016 at 1:28 PM, Jerry Smith (WPT) <
>>>> jdsmith@microsoft.com> wrote:
>>>>
>>>> Summary of issues related to each failure is:
>>>>
>>>>
>>>>
>>>> Persistent-license failures - 1 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, check events
>>>>
>>>>
>>>>
>>>>      This is being internally investigated at Google for a Chrome OS
>>>> fix.
>>>>
>>>>
>>>>
>>>> Temporary session failures - 2 Failures
>>>>
>>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>>> of total)
>>>>
>>>> drm, cannot load persistent license into temporary session
>>>>
>>>>
>>>>
>>>>      This is being worked under https://github.com/w3c/web-platform-tests/issues/4027
>>>> assigned to mwatson2
>>>>
>>>>
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>>>> 0.33% of total)
>>>>
>>>> drm, temporary, mp4, playback, check events
>>>>
>>>>
>>>>
>>>>      This is actually a new failure that didn’t occur on the 10/24/2016
>>>> pass.  Can anyone comment on the test?
>>>>
>>>>
>>>>
>>>> Temporary session readyState failures (listed at risk, under
>>>> discussion) - 2 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>>> readyState
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>>
>>>>
>>>>
>>>>      This issue is dependent on some spec clarifications in these
>>>> issues:
>>>>
>>>>      https://github.com/w3c/encrypted-media/issues/338 - closed
>>>>
>>>>      https://github.com/w3c/encrypted-media/issues/339 - closed
>>>>
>>>>      https://github.com/w3c/encrypted-media/issues/336 - in review
>>>>
>>>>
>>>>
>>>>      I’m not sure if we have an issue opened to modify the test once
>>>> the spec issues are cleared.
>>>>
>>>>
>>>>
>>>> Jerry
>>>>
>>>>
>>>>
>>>> *From:* Jerry Smith (WPT) [mailto:jdsmith@microsoft.com]
>>>> *Sent:* Thursday, October 27, 2016 4:40 PM
>>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>; David Dorwin <
>>>> ddorwin@google.com>; Mark Watson <watsonm@netflix.com>; Matt Wolenetz <
>>>> wolenetz@google.com>; 'public-html-media@w3.org' <
>>>> public-html-media@w3.org>
>>>> *Subject:* EME Test Status Update - 10-27-16
>>>>
>>>>
>>>>
>>>> I’ve posted updated test status reports to reflect a few bug fixes and
>>>> also to add Chrome OS results.
>>>>
>>>>
>>>>
>>>> Reports:
>>>>
>>>>
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/all.html
>>>>
>>>> o   *Test files: 97; Total subtests: 306*
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/complete-f
>>>> ails.html
>>>>
>>>> o   *Completely failed files: 16; Completely failed subtests: 16;
>>>> Failure level: 16/306 (5.23%)*
>>>>
>>>> ·        http://w3c.github.io/test-results/encrypted-media/less-than-
>>>> 2.html
>>>>
>>>> o   *Test files without 2 passes: 28; Subtests without 2 passes: 40;
>>>> Failure level: 40/306 (13.07%)*
>>>>
>>>> Comments:
>>>>
>>>>
>>>>
>>>> -   Complete-fails dropped from 21 to 16, and Less-Than-2 dropped from
>>>> 47 to 40 subtest failures.
>>>>
>>>> -   Persistent-license failures dropped from 5 to 1 with the addition
>>>> of Chrome on Chrome OS results.
>>>>
>>>> -   One temporary session test failed that was previously passing:
>>>>
>>>> o   /encrypted-media/drm-mp4-playback-temporary-events.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> o   drm, temporary, mp4, playback, check events
>>>>
>>>> -   There are only 5 tests in complete-fails that we are still working
>>>> on for our conformance record.
>>>>
>>>> -   That reflects our revised criteria of having a single passing
>>>> implementation (in addition to broad interoperable experience).
>>>>
>>>> In line with that criteria, I’ve chosen to break down just the
>>>> complete-fails test results for this summary.
>>>>
>>>>
>>>>
>>>> Browser Versions:
>>>>
>>>>
>>>>
>>>> CD56:  Chrome 56.0.2902.0 canary (64-bit)
>>>>
>>>> CO56:  Chrome OS 56.0.2900.0 canary (64-bit)
>>>>
>>>> ED14:  Edge 14, Windows 10.0.14393.351 (64-bit)
>>>>
>>>> FF52:  Firefox 52.0a1 (2016-10-27) (32-bit)
>>>>
>>>>
>>>>
>>>> Summary Groupings;
>>>>
>>>>
>>>>
>>>> Errors (considered bugs, not blocking)
>>>> 0
>>>>
>>>> Clear Key persistent-license (not planned, may make
>>>> non-normative)         5
>>>>
>>>> Clear Key persistent-usage-record (at risk, planned to be
>>>> removed)         3
>>>>
>>>> Clear Key transitions between clear and encrypted (bugs, not
>>>> blocking)     0
>>>>
>>>>
>>>>
>>>> Not Blocking Tests
>>>>  8
>>>>
>>>>
>>>>
>>>> Persistent-license (fixes in work)
>>>>           1
>>>>
>>>> Persistent-usage-record (at risk, planned to be
>>>> removed)                   3
>>>>
>>>> Temporary session edge cases (bugs, not block, fixes in
>>>> work)              2
>>>>
>>>> Temporary session readyState (issues under discussion, at
>>>> risk)            2
>>>>
>>>>
>>>>
>>>> In work or at risk
>>>> 8
>>>>
>>>>
>>>>
>>>> TOTAL
>>>> 16
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Details on failed tests are below.
>>>>
>>>>
>>>>
>>>> Jerry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Error handling (considered bugs, not blocking) - 0 Failures
>>>>
>>>>
>>>>
>>>> Clear key persistent-license tests (not planned to be implemented, may
>>>> make informative note) - 5 Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, destroy and
>>>> acknowledge
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-license-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, check events
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-license.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4playback
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-destroy-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve, playback
>>>> and destroy
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-license.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-license, mp4, playback, retrieve and
>>>> playback
>>>>
>>>>
>>>>
>>>> Clear key persistent-usage-record (not planned to be implemented) - 3
>>>> Failures
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, check events
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4playback
>>>>
>>>> /encrypted-media/clearkey-mp4-playback-retrieve-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> org.w3.clearkey, persistent-usage-record, mp4, playback, retrieve in
>>>> new window
>>>>
>>>>
>>>>
>>>> Clear key transitions between clear and encrypted content (considered
>>>> bugs, not blocking) - 2 Failures
>>>>
>>>>
>>>>
>>>> Persistent-license failures - 6 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-license-events.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-license, mp4, playback, check events
>>>>
>>>>
>>>>
>>>> Persistent-usage-record failures - 3 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record-events.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4, playback, check events
>>>>
>>>> /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4, playback, retrieve in new window
>>>>
>>>> /encrypted-media/drm-mp4-playback-persistent-usage-record.html (1/1,
>>>> 100.00%, 0.33% of total)
>>>>
>>>> drm, persistent-usage-record, mp4playback
>>>>
>>>>
>>>>
>>>> Temporary session failures - 2 Failures
>>>>
>>>> /encrypted-media/drm-temporary-license-type.html (1/1, 100.00%, 0.33%
>>>> of total)
>>>>
>>>> drm, cannot load persistent license into temporary session
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-events.html (1/1, 100.00%,
>>>> 0.33% of total)
>>>>
>>>> drm, temporary, mp4, playback, check events
>>>>
>>>>
>>>>
>>>> Temporary session readyState failures (listed at risk, under
>>>> discussion) - 2 Failures
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential,
>>>> readyState
>>>>
>>>> /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.html
>>>> (1/1, 100.00%, 0.33% of total)
>>>>
>>>> drm, successful playback, temporary, mp4, multiple keys, sequential
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>