Re: Using wpt for CR exit

On 03/29/2016 12:33 AM, Geoffrey Sneddon wrote:
> Hey,
>
> Given as far as I know you've been at least somewhat involved with every
> spec that's used wpt as part of its implementation report, let me throw
> a few questions at you:
>
>   * How's the lack of versioning been dealt with? (e.g., between HTML5
> and tests for features added since HTML5 was forked)

We used wptreport and filtering out test results we were not interested in:
  https://github.com/w3c/test-results/blob/gh-pages/html/filter.js

>   * How's the W3C/WHATWG divergence been dealt with? (For specs for which
> that applies!)

At the time of HTML 5.0, the diff between WHATWG and W3C was pretty much 
the same as the diff between 5.0 and 5.1, so we just filtered out the 
results that were only pertinent to 5.1. Other differences didn't have 
impact on implementations.

I do realize that, while we were able to get away with this approach 
back then, it wouldn't be so easy if an API was making a breaking change 
such as changing the return type of a function.

> Far as I can tell, neither of them really have been dealt with. If I'm
> not misreading the few IRs I opened up, they've all assumed that all
> tests within the respective directories in wpt apply to the spec leaving
> CR. Furthermore, when it comes to links to the testsuite, they seem to
> all just link to that directory in the master branch of wpt, which will
> become increasingly less useful at testing a stable snapshot as time
> progresses!

We indeed didn't make a snapshot of the tests at the time. I guess if 
one wanted to run the old html tests, one could checkout an old WPT 
commit from that time.

> There's also the interesting case of a edit to a REC that ends up being
> first published as a CR, where one would presumably want to add/modify
> tests relative to the testsuite for the REC while excluding tests for
> more recent revisions of the spec to exit CR then.
>
> As an aside: there's been some discussion in the CSS WG about testsuites
> recently (uh, my doing, "whoops") which you might have some interest in,
> some of which deals with issues like testsuites for different versions
> of a spec.

I'll try to catch up on this front. I'm curious about the use cases for 
the old test suites. While I can think of some, I'm worried about the 
added costs for maintaining multiple versions of a test suite. We 
already have such difficulties for maintaining multiple versions of a 
spec...

Philippe

Received on Tuesday, 29 March 2016 12:22:13 UTC