Re: Test Changes Proposal

For the file-parser tests its currently one test per vtt file, and any
assertions are either in the vtt file (as attributes for a `VTTCue`
object, eg: `{"line":0}`) or in cue-counts.json (which just assets
that a specific number of cues were parsed).
That doesn't easily expand into tests for stylesheet or VTTRegion parsing.

On Wed, Feb 15, 2017 at 4:37 PM, Philip Jägenstedt <foolip@google.com> wrote:
> Perhaps a PR converting one test would make it easier to judge what this
> would amount to. I guess that a problem with the current tests is that the
> assertions are completely separate from the VTT file itself, but one benefit
> is that the same VTT file can be shared among multiple tests.
>
> On Wed, Feb 15, 2017 at 6:33 AM Benjamin Schaaf <ben.schaaf@gmail.com>
> wrote:
>>
>> I would change it to a format that includes both a vtt file and some
>> js out of which I can generate a test that adds the vtt file as a
>> track to a video element and passed the track and video to the js (and
>> run it).
>> That means I can write arbitrary assertions alongside the vtt file.
>>
>> On Tue, Feb 14, 2017 at 11:22 PM, Philip Jägenstedt <foolip@google.com>
>> wrote:
>> > Can you elaborate on how you'd change the parser tests? That seems like
>> > the
>> > only significant change. CC Simon in case he already replied about this
>> > before this was forwarded to the list.
>> >
>> >
>> > On Tue, Feb 14, 2017 at 2:43 PM Benjamin Schaaf <ben.schaaf@gmail.com>
>> > wrote:
>> >>
>> >> ---------- Forwarded message ----------
>> >> From: Benjamin Schaaf <ben.schaaf@gmail.com>
>> >> Date: Tue, Feb 14, 2017 at 12:48 PM
>> >> Subject: Test Changes Proposal
>> >> To: David Singer <singer@apple.com>, Simon Pieters <simonp@opera.com>,
>> >> Silvia Pfeiffer <silviapfeiffer1@gmail.com>
>> >>
>> >>
>> >> Hello,
>> >>
>> >> I've put together a proposal for a way to reorganise webvtt's
>> >> web-platform-tests to make writing and reading tests easier as well as
>> >> add categorization.
>> >> I'd like to get input on this before I start working on it.
>> >>
>> >> Thanks,
>> >> Benjamin Schaaf
>> >>
>> >>
>> >> # WebVTT web-platform-tests changes proposal
>> >>
>> >> A lot of the way tests are currently set up is limited in some manner,
>> >> eg.
>> >> `webvtt-file-parsing tests` can only check direct attributes of webvtt
>> >> cues and
>> >> the number of cues generated. Instead of building on top of the current
>> >> tests,
>> >> I'd like to rewrite the way the current tests are run so we can more
>> >> easily
>> >> write tests.
>> >>
>> >> For `webvtt-file-parsing` I'd like to generate tests from a template
>> >> that
>> >> includes a wevtt file and some js assertions given the video object.
>> >>
>> >> For `webvtt-cue-text-parsing-rules` I'd like to change the format to be
>> >> more
>> >> easily readable and clean up buildtests.py (or rewrite).
>> >>
>> >> I think the api tests are generally fine as they are (in terms of the
>> >> way
>> >> tests
>> >> are run).
>> >>
>> >> I haven't dived deep enough into rendering tests to come up with
>> >> anything
>> >> concrete, but there should be a nice way to combine a webvtt file, the
>> >> test file
>> >> and a ref file into a more easily writeable/readable single-file format
>> >> that we
>> >> can generate the rest from.
>> >>
>> >> ## Directory Structure
>> >>
>> >> I'd also like to clean up the directory structure. I'm not sure if any
>> >> deeper
>> >> directories are needed
>> >>
>> >> webvtt/
>> >>   api/
>> >>     VTTCue/
>> >>     VTTRegion/
>> >>   parsing/
>> >>     file-parsing/
>> >>     cue-text-parsing/
>> >>   rendering/
>> >>     TBD
>> >>
>> >> ## Categorization
>> >>
>> >> I propose we categorize tests by keeping track of the category of a
>> >> test
>> >> separately (eg. in a `categories.json` file). We can then use the json
>> >> output of
>> >> the test runner to parse which categories/parts of the spec are well
>> >> supported.
>> >>
>> >> The tests for `2dcontext` do something similar.
>> >>
>> >

Received on Wednesday, 15 February 2017 05:42:29 UTC