Re: TTML2 test suite and implementation report

On Thu, Aug 2, 2018 at 6:28 PM, Cyril Concolato <cconcolato@netflix.com>
wrote:

> Hi all,
>
> As discussed this morning, in order to meet the earliest possible deadline
> for publication of the PR, we need to have the test suite and
> implementation report ready by September 27. Once the test suite is ready,
> we should leave time for people to actually check their implementations
> against it. We have 255 features in TTML2. We agreed to test only the
> additional ones compared to TTML1. This means 255-114 = 141 features. Given
> the volume, I think we nee to give people a week to check all those tests
> (possibly produce images to check interop) and update the report. So the
> test suite should be ready before Sept 20. Do you agree?
>
> How do we organize the development of the test suite:
> - Who can provide tests?
> - How do we avoid duplicating the work?
> - I see that https://github.com/w3c/ttml2-tests contains some tests
> already, in a validation folder. What does it mean? Do we foresee
> "validation" tests and "rendering" tests or are they going to be the same?
>

The tests under the validation folder are for validation processing only,
and not meant for presentation testing. There will be an additional top
level directory called presentation, and perhaps one called transformation,
to contain presentation and transformation tests.

I am actively populating the ttml2-tests repo at this time, focusing
initially on validation tests.


>
> @Glenn, you indicated that you have many tests already:
>  - Can you provide them for August 13?
>

I am actively populating tests on a day by day basis. There is no
requirement for the test suite to be fully populated at the start of CR3;
rather, it should be populated by the CfC to transition to PR.


>  - Out of the new 141 features, how many features are tested by your
> tests? How many feature are missing? Can we distribute the work to create
> the missing one among other members?
>

See the spreadsheet [1] I recently posted, which I am populating actively
at this time (an ongoing process). I am initially focusing on validation
tests, and intend to follow up with presentation tests. I will be initially
populating presentation tests from the tests found under TTPE [2], many of
which exercise multiple features. Once these are populated, I will follow
up with additional presentation tests for areas not adequately covered for
our initial test suite. Note that there are no hard rules about how deep or
wide tests need to be to claim that a feature is tested.

[1]
https://docs.google.com/spreadsheets/d/1nunAY0F0bxL0nZDng2VxmWHv7i2RyAIvKQwDiiVzooc/edit?usp=sharing
[2]
https://github.com/skynav/ttt/tree/master/ttt-ttpe/src/test/resources/com/skynav/ttpe/ttml2