RE: FW: Update on MSE and EME Testing

Persistent-usage-record is at least partially supported by Microsoft Edge and an embedded browser that Mark Watson has identified.  It’s still a work in progress, though time is running short.

Thanks for your info.  We are continuing to work through test issues, so if you encounter any more specifics about tests, please let us know.

I’m pretty sure we are not using your nightly build.  We should do that.

Assuming it, the MSE risk features from my original mail should update to:


-          MSE requirements covered by two implementations:

o   The on{event} attributes

o   Live seekable range

o   Resetting the delaying-the-load event in Edge (this makes most test files timeout but note individual tests may still pass)



-          MSE requirements covered by 1 or less:

o    audioTracks and videoTracks in Firefox

o   TrackDefault (feature marked at-risk)



-          For EME, the two remaining at risk features are likely covered by two implementations, if you check in your “waitingforkey” change:

o   The "persistent-usage-record" session type.

o   Setting the media element's readyState value based on key availability and the "waitingforkey" Event.

§  I see this issue from Joey Parrish and commented on by you:  https://github.com/w3c/encrypted-media/issues/284.  Does this issue capture your main issues with waitingforkey?

o   Support for insecure contexts has been removed from the spec already.


-          For EME, the two remaining at risk features are likely covered by at least two implementations:

o   https://github.com/w3c/web-platform-tests/issues/3430




If I’ve missed anything specific here, please let me know.



Jerry

From: Chris Pearce [mailto:cpearce@mozilla.com]
Sent: Wednesday, August 3, 2016 3:59 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: public-html-media@w3.org; jyavenard@mozilla.com; Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>
Subject: Re: FW: Update on MSE and EME Testing

Hi Jerry,
Regarding EME, we have no plans to implement "persistent-usage-record" sessions. You mention two implementations which are "close to passing", which implementations are those?
We have a patch for waitingforkey that apparently works, we've just not gotten around to cleaning it up and landing it. We hadn't prioritized it until Joey Parish from the Shaka Player complained about it<https://bugzilla.mozilla.org/show_bug.cgi?id=1145011#c1>. And then Shaka Player went and removed their use of "waitingforkey" altogether and complained they couldn't tell when they weren't waiting for a key anymore<https://github.com/w3c/encrypted-media/issues/284>... So I'd like to hear Joey answer my comment in that issue before landing our patch.
So we could land our waitingforkey patch tomorrow if we wanted, but it's not clear to me that the authors who wanted that feature want it in its current form. I wouldn't object if we dropped waitingforkey.

FYI, I am working towards getting Google's Web Platform EME Tests passing in Firefox 51 Nightly. I've fixed a few issues so far. Most of the problems I've found so far are bugs like us not throwing the specified exception code, or parser bugs. I hope to have them passing or debugged in the next few weeks. I will file issues against the tests if I find problems. I've filed issue 3430<https://github.com/w3c/web-platform-tests/issues/3430> so far.

Cheers,
Chris Pearce.

On Thu, Aug 4, 2016 at 2:43 AM, Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>> wrote:
Forwarding this thread to public-html-media@w3.org<mailto:public-html-media@w3.org> with permission.

This thread is about Firefox’s implementation of specific MSE features and their comments on the MSE test suite.

/paulc

From: jyavenard@mozilla.com<mailto:jyavenard@mozilla.com> [mailto:jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>]
Sent: Tuesday, August 2, 2016 10:11 PM

To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Jean-Yves Avenard <jya@mozilla.com<mailto:jya@mozilla.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>
Subject: RE: Update on MSE and EME Testing

Hello

Here they are:
https://github.com/w3c/web-platform-tests/issues/created_by/jyavenard


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Paul Cotton<mailto:Paul.Cotton@microsoft.com>
Sent: Wednesday, 3 August 2016 10:07 PM
To: jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>; Jean-Yves Avenard<mailto:jya@mozilla.com>; Jerry Smith (WPT)<mailto:jdsmith@microsoft.com>
Cc: Chris Pearce<mailto:cpearce@mozilla.com>; Matt Wolenetz<mailto:wolenetz@google.com>; Anthony Jones<mailto:ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org)<mailto:plh@w3.org>
Subject: RE: Update on MSE and EME Testing

> I believe I have opened some bugs on web-platform-tests GitHub ; but no progresses were made.

Could you please provide pointers to those bugs?

/paulc

From: jyavenard@mozilla.com<mailto:jyavenard@mozilla.com> [mailto:jyavenard@mozilla.com]
Sent: Tuesday, August 2, 2016 10:07 PM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Jean-Yves Avenard <jya@mozilla.com<mailto:jya@mozilla.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>
Subject: RE: Update on MSE and EME Testing

Sure.

I believe I have opened some bugs on web-platform-tests GitHub ; but no progresses were made.

Note that we made some modifications in regards to the handling of the duration change and that the coded frame removal algorithm can no longer be interrupted.
The changes were made in our copy of the tests; it is my understanding they will be synced automatically soon.

The bugs tracking those changes are:
https://bugzilla.mozilla.org/show_bug.cgi?id=1286810 (duration change and throwing an error when aborting during a range removal is in progress)
https://bugzilla.mozilla.org/show_bug.cgi?id=1286796 (replacing InvalidAccessError with TypeError)

Kind regards
Jean-Yves


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Paul Cotton<mailto:Paul.Cotton@microsoft.com>
Sent: Wednesday, 3 August 2016 9:33 PM
To: Jean-Yves Avenard<mailto:jya@mozilla.com>; Jerry Smith (WPT)<mailto:jdsmith@microsoft.com>
Cc: Chris Pearce<mailto:cpearce@mozilla.com>; Matt Wolenetz<mailto:wolenetz@google.com>; Anthony Jones<mailto:ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org)<mailto:plh@w3.org>
Subject: RE: Update on MSE and EME Testing

+ Philippe

Can I please have permission to post this thread to public-html-media@w3.org<mailto:public-html-media@w3.org> so that these results and possible problems with the MSE test suite are broadly known.

/paulc

From: Jean-Yves Avenard [mailto:jya@mozilla.com]
Sent: Wednesday, August 3, 2016 3:55 AM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>
Subject: Re: Update on MSE and EME Testing

Hello Jerry

On Wed, Aug 3, 2016 at 3:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
Hi Chris and Jean-Yves,

I previously had some exchanges with you on MSE and EME testing.  The HTML Media Extensions group is in the stretch run on both specs, and I have a few feature questions to confirm with you.  Matt Wolenetz from Google is on this thread as well and may have follow on questions to mine.

MSE:

There’s an MSE test report here:  http://w3c.github.io/test-results/media-source/all.html.  It currently shows a number of interface tests that seem to be timing out.

We do see tests intermittently timing out.
You'll find here the expected results and tests skipped for those media-source test.
http://hg.mozilla.org/mozilla-central/file/tip/testing/web-platform/meta/media-source

We've worked hard to pass all the media-source webref tests. Thorough investigations were performed when we didn't pass some tests, and i can pretty confidently say that other than those related to features we do not support (e.g. tracks), every tests actually failing or timing out are actually a problem in the test.
For the tests disabled, you'll find the bugzilla number on why the test was disabled.
Those are typically intermittent timeout or failures.
Unfortunately, the web-platform-test architecture doesn't allow to add a comment for the tests we didn't disabled, but instead expected failures

We have come to the conclusion that those tests often just verify Chrome specific behaviour (and verified as chrome being the only browser passing those tests).
One example is the buffered range test. The mp4 sample used in all the tests do not start at 0. The first PTS is actually 0.096 (only the dts is 0). As such, the buffered range start shouldn't be 0.
Many tests actually first check that the buffered range start at 0 and abort early, so we get an early failure.
For some intermittent failures or timeout, those are the seek tests or playback progress tests. Often those tests fail because they expect media events to be received in a particular order. The timeupdate event in particular often gets in the way and cause the tests to fail.

Ignoring that, we still lack two passing implementations on several features.  You can see those here:  http://w3c.github.io/test-results/media-source/less-than-2.html.  The feature gaps I previously sent you still show as lacking implementations in the latest test results:


I'm not sure how the tests are run, but that's certainly not what I'm seeing from a nightly build here :(


1.       The on{event} attributes
done

2.       Live seekable range
done

3.       audioTracks and videoTracks in Firefox
won't do

4.       Resetting the delaying-the-load event in Edge (this makes most test files timeout but note individual tests may still pass)
done

5.       TrackDefault (feature marked at-risk)
won't do.

Jean-Yves:  I believe you planned to implement the eventHandler attributes and live seekable range.  What build should we be testing to use them?  And are audioTracks/videoTracks and TrackDefaults still not planned?

They will be in our Nightly build after July 16th 2016.


On MSE features identified as at risk in the spec:


1.       appendStream is planned to be removed (since it is not current).

2.       VideoPlaybackQuality is proposed to move into an extension spec.  I believe only Microsoft has pushed hard to retain this, and appears to be the only browser passing all tests (though only 2 of 20 tests lack two passing, both on totalFrameDelay).

Hmm... we do support VideoPlaybackQuality, I don't believe we support totalFrameDelay however.

3.       The TrackDefault object and its related objects are still TBD. (Chrome may have this behind a flag – Matt please confirm).


we don't support track related object/attributes. I'm not sure if we have any plans to do so.
We do have everything in place to support it, just never given it any priority on our tasks list.
Hope this help
Jean-Yves