Re: Update on MSE and EME Testing

>
>
>    - There is a bug in util/utils.js in getSupportedKeysystem() which
>    returns com.widevine.alpha for Edge, meaning that playback tests on Edge
>    all fail. I presume you fixed this before generating the results.
>
> Mark has a fix for this in
https://github.com/w3c/web-platform-tests/pull/3424/files#diff-c5c73a9a657afac9f2d88b36c6348243
if someone wants to land that separately.

I plan to pull down the older versions of Chrome and Firefox soon, but
> wanted to be sure the deltas between the two make sense.  Comments are
> welcome.
>
Jerry, the two failures in CH52 where CH54 passes are fine:

   - /encrypted-media/Google/encrypted-media-async-setcert-with-gc.html
   fails in CH52 because we changed the behavior and test to be spec compliant.
   - /encrypted-media/clearkey-mp4-playback-temporary-events.html fails
   because of a bug that we fixed when it was identified by this test.


I don't understand some of the output. For example,
http://www.w3c-test.org/encrypted-media/Google/encrypted-media-keystatuses.html
has no result listed for several browsers. It passes in the latest Chrome
but there is no entry. Also, TIMEOUT seems to appear inconsistently in some
header rows. The persistent-usage-record tests don't even have results.
Does this mean they are new / not run?

On Fri, Aug 5, 2016 at 2:14 PM, Mark Watson <watsonm@netflix.com> wrote:

> Jerry,
>
> There are a few things which need to be fixed up before we can really read
> anything into the test results:
>
>    - There is a bug in util/utils.js in getSupportedKeysystem() which
>    returns com.widevine.alpha for Edge, meaning that playback tests on Edge
>    all fail. I presume you fixed this before generating the results.
>    - The "less than 2" script should not consider two versions of Chrome
>    or two versions of Firefox as 2 versions passing
>    - The test names for the drm tests include the keysystem. So, when you
>    have it working with appropriate keysystems on each browser, they will not
>    match up correctly on the test-results page. We'll need to remove the
>    keysystem name from the test name and just put "drm"
>    - The test names for the polyfill tests should probably start
>    "Polyfill" for readability
>
> ...Mark
>
> On Fri, Aug 5, 2016 at 1:44 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
>> I’ve made an initial posting of test results on a few browsers (Edge, two
>> versions of Chrome and two also of Firefox).  It’s a bit messy, because it
>> includes Google subfolder tests for Clear-Key (dups) and also tests with
>> and without some polyfills Mark has been developing.
>>
>>
>>
>> You can see current EME results using these links:
>>
>>
>>
>> •                     http://w3c.github.io/test-resu
>> lts/encrypted-media/all.html
>>
>> •                     http://w3c.github.io/test-resu
>> lts/encrypted-media/complete-fails.html
>>
>> •                     http://w3c.github.io/test-resu
>> lts/encrypted-media/less-than-2.html
>>
>>
>>
>> I plan to pull down the older versions of Chrome and Firefox soon, but
>> wanted to be sure the deltas between the two make sense.  Comments are
>> welcome.
>>
>>
>>
>> For convenience, here are the MSE links:
>>
>>
>>
>> •                     http://w3c.github.io/test-resu
>> lts/media-source/all.html
>>
>> •                     http://w3c.github.io/test-resu
>> lts/media-source/complete-fails.html
>>
>> •                     http://w3c.github.io/test-resu
>> lts/media-source/less-than-2.html
>>
>>
>>
>> Jerry
>>
>>
>>
>> *From:* Mark Watson [mailto:watsonm@netflix.com]
>> *Sent:* Friday, August 5, 2016 12:44 PM
>> *To:* Henri Sivonen <hsivonen@hsivonen.fi>
>> *Cc:* Chris Pearce <cpearce@mozilla.com>; Jerry Smith (WPT) <
>> jdsmith@microsoft.com>; public-html-media@w3.org; jyavenard@mozilla.com;
>> Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
>> wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le
>> Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com
>> >
>> *Subject:* Re: Update on MSE and EME Testing
>>
>>
>>
>> We have been working on additional tests for the Encrypted Media
>> Extensions specification.
>>
>>
>>
>> Unfortunately, it has been very difficult to get the tests we have been
>> working to work, due to lack of the necessary server support and
>> uncertainty as to what is and is not implemented in  browsers. At this
>> stage, therefore, we do not yet have DRM tests for persistent-license or
>> persistent-usage-record session types.
>>
>>
>>
>> There are two outstanding pull requests:
>>
>> - https://github.com/w3c/web-platform-tests/pull/3424 adds support for
>> persistent-license (clearkey)
>>
>> - https://github.com/w3c/web-platform-tests/pull/3417 adds a
>> waitingforkey test
>>
>>
>>
>> I expect we will make further progress over the coming week (I will be
>> out, but Sukmal will have some time to work on this).
>>
>>
>>
>> ...Mark
>>
>>
>>
>>
>>
>> On Fri, Aug 5, 2016 at 7:50 AM, Mark Watson <watsonm@netflix.com> wrote:
>>
>>
>>
>>
>>
>> On Fri, Aug 5, 2016 at 1:20 AM, Henri Sivonen <hsivonen@hsivonen.fi>
>> wrote:
>>
>> On Thu, Aug 4, 2016 at 7:26 AM, Mark Watson <watsonm@netflix.com> wrote:
>> > I believe that persistent-usage-record functionality works with Firefox
>> /
>> > Primetime, with some rough edges on the Primetime side. However, we do
>> not
>> > have a Primetime server for our testing, so we cannot verify this.
>>
>> While some initial steps (that I've referred to previously on this
>> list) were taken towards persistent-usage-record -like functionality
>> on the Firefox side, those steps never reached the point of supporting
>> the persistent-usage-record API. As Chris said, we have no plans to
>> implement "persistent-usage-record" sessions.
>>
>> > The Edge and CastTV implementations are close to compliance, I believe,
>> and we are working on tests that will demonstrate that.
>>
>> Do those implementations write to plain disk-like storage on CDM
>> shutdown or do they use some form of tamper-resistant storage during
>> playback? Which Key System does CastTV use? Was the text in the spec
>> alone sufficiently detailed to reach interop?
>>
>>
>>
>> ​I don't know the implementation details. They are both PlayReady. They
>> don't reach interop. When I say "close to compliance", I mean that with a
>> bit of JS help you can patch up the non-compliances. At least, that is my
>> hope at present: we don't have everything working yet.
>>
>>
>>
>> ...Mark
>>
>>
>>
>>
>> --
>> Henri Sivonen
>> hsivonen@hsivonen.fi
>> https://hsivonen.fi/
>>
>>
>>
>>
>>
>
>