Re: Test Changes Proposal

I've converted the 'omitted-hours' test to what I'm suggesting:
https://gist.github.com/BenjaminSchaaf/0b32042c609d5d5d7c837d2247e391ec
I've included the html and vtt files that would be generated.

First line is the name of the test, followed by any number of html headers.
Then comes the js portion of the test with access to the video, track
and cues objects.
Then comes the vtt portion which is automatically added/loaded.

We could even leave the current tests intact, though I'd prefer to
convert them so everything is in a uniform format.

On Wed, Feb 15, 2017 at 4:54 PM, Philip Jägenstedt <foolip@google.com> wrote:
> That is true, neither of those have payloads where the expectations can be
> put, although for the existing parser tests I assume that only really works
> for the key:value things trailing the timing, and not parsing of the cue
> text itself.
>
> Maybe a new test for stylesheet parsing using your suggested approach?
>
> On Wed, Feb 15, 2017 at 12:41 PM Benjamin Schaaf <ben.schaaf@gmail.com>
> wrote:
>>
>> For the file-parser tests its currently one test per vtt file, and any
>> assertions are either in the vtt file (as attributes for a `VTTCue`
>> object, eg: `{"line":0}`) or in cue-counts.json (which just assets
>> that a specific number of cues were parsed).
>> That doesn't easily expand into tests for stylesheet or VTTRegion parsing.
>>
>> On Wed, Feb 15, 2017 at 4:37 PM, Philip Jägenstedt <foolip@google.com>
>> wrote:
>> > Perhaps a PR converting one test would make it easier to judge what this
>> > would amount to. I guess that a problem with the current tests is that
>> > the
>> > assertions are completely separate from the VTT file itself, but one
>> > benefit
>> > is that the same VTT file can be shared among multiple tests.
>> >
>> > On Wed, Feb 15, 2017 at 6:33 AM Benjamin Schaaf <ben.schaaf@gmail.com>
>> > wrote:
>> >>
>> >> I would change it to a format that includes both a vtt file and some
>> >> js out of which I can generate a test that adds the vtt file as a
>> >> track to a video element and passed the track and video to the js (and
>> >> run it).
>> >> That means I can write arbitrary assertions alongside the vtt file.
>> >>
>> >> On Tue, Feb 14, 2017 at 11:22 PM, Philip Jägenstedt <foolip@google.com>
>> >> wrote:
>> >> > Can you elaborate on how you'd change the parser tests? That seems
>> >> > like
>> >> > the
>> >> > only significant change. CC Simon in case he already replied about
>> >> > this
>> >> > before this was forwarded to the list.
>> >> >
>> >> >
>> >> > On Tue, Feb 14, 2017 at 2:43 PM Benjamin Schaaf
>> >> > <ben.schaaf@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> ---------- Forwarded message ----------
>> >> >> From: Benjamin Schaaf <ben.schaaf@gmail.com>
>> >> >> Date: Tue, Feb 14, 2017 at 12:48 PM
>> >> >> Subject: Test Changes Proposal
>> >> >> To: David Singer <singer@apple.com>, Simon Pieters
>> >> >> <simonp@opera.com>,
>> >> >> Silvia Pfeiffer <silviapfeiffer1@gmail.com>
>> >> >>
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> I've put together a proposal for a way to reorganise webvtt's
>> >> >> web-platform-tests to make writing and reading tests easier as well
>> >> >> as
>> >> >> add categorization.
>> >> >> I'd like to get input on this before I start working on it.
>> >> >>
>> >> >> Thanks,
>> >> >> Benjamin Schaaf
>> >> >>
>> >> >>
>> >> >> # WebVTT web-platform-tests changes proposal
>> >> >>
>> >> >> A lot of the way tests are currently set up is limited in some
>> >> >> manner,
>> >> >> eg.
>> >> >> `webvtt-file-parsing tests` can only check direct attributes of
>> >> >> webvtt
>> >> >> cues and
>> >> >> the number of cues generated. Instead of building on top of the
>> >> >> current
>> >> >> tests,
>> >> >> I'd like to rewrite the way the current tests are run so we can more
>> >> >> easily
>> >> >> write tests.
>> >> >>
>> >> >> For `webvtt-file-parsing` I'd like to generate tests from a template
>> >> >> that
>> >> >> includes a wevtt file and some js assertions given the video object.
>> >> >>
>> >> >> For `webvtt-cue-text-parsing-rules` I'd like to change the format to
>> >> >> be
>> >> >> more
>> >> >> easily readable and clean up buildtests.py (or rewrite).
>> >> >>
>> >> >> I think the api tests are generally fine as they are (in terms of
>> >> >> the
>> >> >> way
>> >> >> tests
>> >> >> are run).
>> >> >>
>> >> >> I haven't dived deep enough into rendering tests to come up with
>> >> >> anything
>> >> >> concrete, but there should be a nice way to combine a webvtt file,
>> >> >> the
>> >> >> test file
>> >> >> and a ref file into a more easily writeable/readable single-file
>> >> >> format
>> >> >> that we
>> >> >> can generate the rest from.
>> >> >>
>> >> >> ## Directory Structure
>> >> >>
>> >> >> I'd also like to clean up the directory structure. I'm not sure if
>> >> >> any
>> >> >> deeper
>> >> >> directories are needed
>> >> >>
>> >> >> webvtt/
>> >> >>   api/
>> >> >>     VTTCue/
>> >> >>     VTTRegion/
>> >> >>   parsing/
>> >> >>     file-parsing/
>> >> >>     cue-text-parsing/
>> >> >>   rendering/
>> >> >>     TBD
>> >> >>
>> >> >> ## Categorization
>> >> >>
>> >> >> I propose we categorize tests by keeping track of the category of a
>> >> >> test
>> >> >> separately (eg. in a `categories.json` file). We can then use the
>> >> >> json
>> >> >> output of
>> >> >> the test runner to parse which categories/parts of the spec are well
>> >> >> supported.
>> >> >>
>> >> >> The tests for `2dcontext` do something similar.
>> >> >>
>> >> >

Received on Thursday, 23 February 2017 03:32:58 UTC