FW: Update on MSE and EME Testing

Forwarding this thread to public-html-media@w3.org<mailto:public-html-media@w3.org> with permission.

This thread is about Firefox’s implementation of specific MSE features and their comments on the MSE test suite.

/paulc

From: jyavenard@mozilla.com [mailto:jyavenard@mozilla.com]
Sent: Tuesday, August 2, 2016 10:11 PM
To: Paul Cotton <Paul.Cotton@microsoft.com>; Jean-Yves Avenard <jya@mozilla.com>; Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: Chris Pearce <cpearce@mozilla.com>; Matt Wolenetz <wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>
Subject: RE: Update on MSE and EME Testing

Hello

Here they are:
https://github.com/w3c/web-platform-tests/issues/created_by/jyavenard


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Paul Cotton<mailto:Paul.Cotton@microsoft.com>
Sent: Wednesday, 3 August 2016 10:07 PM
To: jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>; Jean-Yves Avenard<mailto:jya@mozilla.com>; Jerry Smith (WPT)<mailto:jdsmith@microsoft.com>
Cc: Chris Pearce<mailto:cpearce@mozilla.com>; Matt Wolenetz<mailto:wolenetz@google.com>; Anthony Jones<mailto:ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org)<mailto:plh@w3.org>
Subject: RE: Update on MSE and EME Testing

> I believe I have opened some bugs on web-platform-tests GitHub ; but no progresses were made.

Could you please provide pointers to those bugs?

/paulc

From: jyavenard@mozilla.com<mailto:jyavenard@mozilla.com> [mailto:jyavenard@mozilla.com]
Sent: Tuesday, August 2, 2016 10:07 PM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Jean-Yves Avenard <jya@mozilla.com<mailto:jya@mozilla.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>
Subject: RE: Update on MSE and EME Testing

Sure.

I believe I have opened some bugs on web-platform-tests GitHub ; but no progresses were made.

Note that we made some modifications in regards to the handling of the duration change and that the coded frame removal algorithm can no longer be interrupted.
The changes were made in our copy of the tests; it is my understanding they will be synced automatically soon.

The bugs tracking those changes are:
https://bugzilla.mozilla.org/show_bug.cgi?id=1286810 (duration change and throwing an error when aborting during a range removal is in progress)
https://bugzilla.mozilla.org/show_bug.cgi?id=1286796 (replacing InvalidAccessError with TypeError)

Kind regards
Jean-Yves


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Paul Cotton<mailto:Paul.Cotton@microsoft.com>
Sent: Wednesday, 3 August 2016 9:33 PM
To: Jean-Yves Avenard<mailto:jya@mozilla.com>; Jerry Smith (WPT)<mailto:jdsmith@microsoft.com>
Cc: Chris Pearce<mailto:cpearce@mozilla.com>; Matt Wolenetz<mailto:wolenetz@google.com>; Anthony Jones<mailto:ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org)<mailto:plh@w3.org>
Subject: RE: Update on MSE and EME Testing

+ Philippe

Can I please have permission to post this thread to public-html-media@w3.org<mailto:public-html-media@w3.org> so that these results and possible problems with the MSE test suite are broadly known.

/paulc

From: Jean-Yves Avenard [mailto:jya@mozilla.com]
Sent: Wednesday, August 3, 2016 3:55 AM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>
Subject: Re: Update on MSE and EME Testing

Hello Jerry

On Wed, Aug 3, 2016 at 3:24 AM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
Hi Chris and Jean-Yves,

I previously had some exchanges with you on MSE and EME testing.  The HTML Media Extensions group is in the stretch run on both specs, and I have a few feature questions to confirm with you.  Matt Wolenetz from Google is on this thread as well and may have follow on questions to mine.

MSE:

There’s an MSE test report here:  http://w3c.github.io/test-results/media-source/all.html.  It currently shows a number of interface tests that seem to be timing out.

We do see tests intermittently timing out.
You'll find here the expected results and tests skipped for those media-source test.
http://hg.mozilla.org/mozilla-central/file/tip/testing/web-platform/meta/media-source

We've worked hard to pass all the media-source webref tests. Thorough investigations were performed when we didn't pass some tests, and i can pretty confidently say that other than those related to features we do not support (e.g. tracks), every tests actually failing or timing out are actually a problem in the test.
For the tests disabled, you'll find the bugzilla number on why the test was disabled.
Those are typically intermittent timeout or failures.
Unfortunately, the web-platform-test architecture doesn't allow to add a comment for the tests we didn't disabled, but instead expected failures

We have come to the conclusion that those tests often just verify Chrome specific behaviour (and verified as chrome being the only browser passing those tests).
One example is the buffered range test. The mp4 sample used in all the tests do not start at 0. The first PTS is actually 0.096 (only the dts is 0). As such, the buffered range start shouldn't be 0.
Many tests actually first check that the buffered range start at 0 and abort early, so we get an early failure.
For some intermittent failures or timeout, those are the seek tests or playback progress tests. Often those tests fail because they expect media events to be received in a particular order. The timeupdate event in particular often gets in the way and cause the tests to fail.

Ignoring that, we still lack two passing implementations on several features.  You can see those here:  http://w3c.github.io/test-results/media-source/less-than-2.html.  The feature gaps I previously sent you still show as lacking implementations in the latest test results:


I'm not sure how the tests are run, but that's certainly not what I'm seeing from a nightly build here :(


1.       The on{event} attributes
done

2.       Live seekable range
done

3.       audioTracks and videoTracks in Firefox
won't do

4.       Resetting the delaying-the-load event in Edge (this makes most test files timeout but note individual tests may still pass)
done

5.       TrackDefault (feature marked at-risk)
won't do.

Jean-Yves:  I believe you planned to implement the eventHandler attributes and live seekable range.  What build should we be testing to use them?  And are audioTracks/videoTracks and TrackDefaults still not planned?

They will be in our Nightly build after July 16th 2016.


On MSE features identified as at risk in the spec:


1.       appendStream is planned to be removed (since it is not current).

2.       VideoPlaybackQuality is proposed to move into an extension spec.  I believe only Microsoft has pushed hard to retain this, and appears to be the only browser passing all tests (though only 2 of 20 tests lack two passing, both on totalFrameDelay).

Hmm... we do support VideoPlaybackQuality, I don't believe we support totalFrameDelay however.

3.       The TrackDefault object and its related objects are still TBD. (Chrome may have this behind a flag – Matt please confirm).


we don't support track related object/attributes. I'm not sure if we have any plans to do so.
We do have everything in place to support it, just never given it any priority on our tasks list.
Hope this help
Jean-Yves