Re: Update on MSE and EME Testing

Chris - note that our plan is to replace the Google tests with the generic
ones Sukhmal and I have been working on. This provides for easy creation of
multiple versions of each test, for clearKey, DRM, MP4, WebM as appropriate.


On Aug 3, 2016, at 4:49 PM, Jerry Smith (WPT) <jdsmith@microsoft.com> wrote:

Persistent-usage-record is at least partially supported by Microsoft Edge
and an embedded browser that Mark Watson has identified.  It’s still a work
in progress, though time is running short.


I doubt that persistent-license is in much better shape than
persistent-usage-record and it is not a 'feature at risk'. I feel we are at
a point where usually the spec would stay at CR until compliance improved.



Thanks for your info.  We are continuing to work through test issues, so if
you encounter any more specifics about tests, please let us know.



I’m pretty sure we are not using your nightly build.  We should do that.



Assuming it, the MSE risk features from my original mail should update to:



-          MSE requirements covered by two implementations:

o   The on{event} attributes

o   Live seekable range

o   Resetting the delaying-the-load event in Edge (this makes most test
files timeout but note individual tests may still pass)



-          MSE requirements covered by 1 or less:

o    audioTracks and videoTracks in Firefox

o   TrackDefault (feature marked at-risk)



-          For EME, the two remaining at risk features are likely covered
by two implementations, if you check in your “waitingforkey” change:

o   The "persistent-usage-record" session type.

o   Setting the media element's readyState value based on key availability
and the "waitingforkey" Event.

§  I see this issue from Joey Parrish and commented on by you:
https://github.com/w3c/encrypted-media/issues/284.  Does this issue capture
your main issues with waitingforkey?

o   Support for insecure contexts has been removed from the spec already.

-          For EME, the two remaining at risk features are likely covered
by at least two implementations:

o   https://github.com/w3c/web-platform-tests/issues/3430



If I’ve missed anything specific here, please let me know.



Jerry



*From:* Chris Pearce [mailto:cpearce@mozilla.com <cpearce@mozilla.com>]
*Sent:* Wednesday, August 3, 2016 3:59 PM
*To:* Jerry Smith (WPT) <jdsmith@microsoft.com>
*Cc:* public-html-media@w3.org; jyavenard@mozilla.com; Matthew Wolenetz <
wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Anthony
Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>;
Paul Cotton <Paul.Cotton@microsoft.com>
*Subject:* Re: FW: Update on MSE and EME Testing



Hi Jerry,

Regarding EME, we have no plans to implement "persistent-usage-record"
sessions. You mention two implementations which are "close to passing",
which implementations are those?

We have a patch for waitingforkey that apparently works, we've just not
gotten around to cleaning it up and landing it. We hadn't prioritized it
until Joey Parish from the Shaka Player complained about it
<https://bugzilla.mozilla.org/show_bug.cgi?id=1145011#c1>. And then Shaka
Player went and removed their use of "waitingforkey" altogether and complained
they couldn't tell when they weren't waiting for a key anymore
<https://github.com/w3c/encrypted-media/issues/284>... So I'd like to hear
Joey answer my comment in that issue before landing our patch.

So we could land our waitingforkey patch tomorrow if we wanted, but it's
not clear to me that the authors who wanted that feature want it in its
current form. I wouldn't object if we dropped waitingforkey.



FYI, I am working towards getting Google's Web Platform EME Tests passing
in Firefox 51 Nightly. I've fixed a few issues so far. Most of the problems
I've found so far are bugs like us not throwing the specified exception
code, or parser bugs. I hope to have them passing or debugged in the next
few weeks. I will file issues against the tests if I find problems. I've
filed issue 3430 <https://github.com/w3c/web-platform-tests/issues/3430> so
far.



Cheers,

Chris Pearce.



On Thu, Aug 4, 2016 at 2:43 AM, Paul Cotton <Paul.Cotton@microsoft.com>
wrote:

Forwarding this thread to public-html-media@w3.org with permission.



This thread is about Firefox’s implementation of specific MSE features and
their comments on the MSE test suite.



/paulc



*From:* jyavenard@mozilla.com [mailto:jyavenard@mozilla.com]
*Sent:* Tuesday, August 2, 2016 10:11 PM


*To:* Paul Cotton <Paul.Cotton@microsoft.com>; Jean-Yves Avenard <
jya@mozilla.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>
*Cc:* Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz <wolenetz@google.com>;
Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <
plh@w3.org>
*Subject:* RE: Update on MSE and EME Testing



Hello



Here they are:

https://github.com/w3c/web-platform-tests/issues/created_by/jyavenard



Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
10



*From: *Paul Cotton <Paul.Cotton@microsoft.com>
*Sent: *Wednesday, 3 August 2016 10:07 PM
*To: *jyavenard@mozilla.com; Jean-Yves Avenard <jya@mozilla.com>; Jerry
Smith (WPT) <jdsmith@microsoft.com>
*Cc: *Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz
<wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le
Hegaret (plh@w3.org) <plh@w3.org>
*Subject: *RE: Update on MSE and EME Testing



> I believe I have opened some bugs on web-platform-tests GitHub ; but no
progresses were made.



Could you please provide pointers to those bugs?



/paulc



*From:* jyavenard@mozilla.com [mailto:jyavenard@mozilla.com
<jyavenard@mozilla.com>]
*Sent:* Tuesday, August 2, 2016 10:07 PM
*To:* Paul Cotton <Paul.Cotton@microsoft.com>; Jean-Yves Avenard <
jya@mozilla.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>
*Cc:* Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz <wolenetz@google.com>;
Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <
plh@w3.org>
*Subject:* RE: Update on MSE and EME Testing



Sure.



I believe I have opened some bugs on web-platform-tests GitHub ; but no
progresses were made.



Note that we made some modifications in regards to the handling of the
duration change and that the coded frame removal algorithm can no longer be
interrupted.

The changes were made in our copy of the tests; it is my understanding they
will be synced automatically soon.



The bugs tracking those changes are:

https://bugzilla.mozilla.org/show_bug.cgi?id=1286810 (duration change and
throwing an error when aborting during a range removal is in progress)

https://bugzilla.mozilla.org/show_bug.cgi?id=1286796 (replacing
InvalidAccessError with TypeError)



Kind regards

Jean-Yves





Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
10



*From: *Paul Cotton <Paul.Cotton@microsoft.com>
*Sent: *Wednesday, 3 August 2016 9:33 PM
*To: *Jean-Yves Avenard <jya@mozilla.com>; Jerry Smith (WPT)
<jdsmith@microsoft.com>
*Cc: *Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz
<wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le
Hegaret (plh@w3.org) <plh@w3.org>
*Subject: *RE: Update on MSE and EME Testing



+ Philippe



Can I please have permission to post this thread to public-html-media@w3.org
so that these results and possible problems with the MSE test suite are
broadly known.



/paulc



*From:* Jean-Yves Avenard [mailto:jya@mozilla.com <jya@mozilla.com>]
*Sent:* Wednesday, August 3, 2016 3:55 AM
*To:* Jerry Smith (WPT) <jdsmith@microsoft.com>
*Cc:* Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz <wolenetz@google.com>;
Paul Cotton <Paul.Cotton@microsoft.com>; Anthony Jones <ajones@mozilla.com>
*Subject:* Re: Update on MSE and EME Testing



Hello Jerry



On Wed, Aug 3, 2016 at 3:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

Hi Chris and Jean-Yves,



I previously had some exchanges with you on MSE and EME testing.  The HTML
Media Extensions group is in the stretch run on both specs, and I have a
few feature questions to confirm with you.  Matt Wolenetz from Google is on
this thread as well and may have follow on questions to mine.



*MSE:*



There’s an MSE test report here:
http://w3c.github.io/test-results/media-source/all.html.  It currently
shows a number of interface tests that seem to be timing out.



We do see tests intermittently timing out.

You'll find here the expected results and tests skipped for those
media-source test.
http://hg.mozilla.org/mozilla-central/file/tip/testing/web-platform/meta/media-source

We've worked hard to pass all the media-source webref tests. Thorough
investigations were performed when we didn't pass some tests, and i can
pretty confidently say that other than those related to features we do not
support (e.g. tracks), every tests actually failing or timing out are
actually a problem in the test.

For the tests disabled, you'll find the bugzilla number on why the test was
disabled.

Those are typically intermittent timeout or failures.

Unfortunately, the web-platform-test architecture doesn't allow to add a
comment for the tests we didn't disabled, but instead expected failures



We have come to the conclusion that those tests often just verify Chrome
specific behaviour (and verified as chrome being the only browser passing
those tests).

One example is the buffered range test. The mp4 sample used in all the
tests do not start at 0. The first PTS is actually 0.096 (only the dts is
0). As such, the buffered range start shouldn't be 0.

Many tests actually first check that the buffered range start at 0 and
abort early, so we get an early failure.

For some intermittent failures or timeout, those are the seek tests or
playback progress tests. Often those tests fail because they expect media
events to be received in a particular order. The timeupdate event in
particular often gets in the way and cause the tests to fail.



Ignoring that, we still lack two passing implementations on several
features.  You can see those here:
http://w3c.github.io/test-results/media-source/less-than-2.html.  The
feature gaps I previously sent you still show as lacking implementations in
the latest test results:





I'm not sure how the tests are run, but that's certainly not what I'm
seeing from a nightly build here :(


1.       The on{event} attributes

done

2.       Live seekable range

done

3.       audioTracks and videoTracks in Firefox

won't do

4.       Resetting the delaying-the-load event in Edge (this makes most
test files timeout but note individual tests may still pass)

done

5.       TrackDefault (feature marked at-risk)

won't do.



*Jean-Yves:*  I believe you planned to implement the eventHandler
attributes and live seekable range.  What build should we be testing to use
them?  And are audioTracks/videoTracks and TrackDefaults still not planned?



They will be in our Nightly build after July 16th 2016.




On MSE features identified as at risk in the spec:



1.       appendStream is planned to be removed (since it is not current).

2.       VideoPlaybackQuality is proposed to move into an extension spec.
I believe only Microsoft has pushed hard to retain this, and appears to be
the only browser passing all tests (though only 2 of 20 tests lack two
passing, both on totalFrameDelay).



Hmm... we do support VideoPlaybackQuality, I don't believe we support
totalFrameDelay however.

3.       The TrackDefault object and its related objects are still TBD.
(Chrome may have this behind a flag – Matt please confirm).





we don't support track related object/attributes. I'm not sure if we have
any plans to do so.

We do have everything in place to support it, just never given it any
priority on our tasks list.

Hope this help

Jean-Yves