Re: Importing W3C's test suite into implementor's test suites.

On Thu, Mar 21, 2013 at 7:28 AM, Tobie Langel <tobie@w3.org> wrote:

> On Thursday, March 21, 2013 at 2:11 PM, James Graham wrote:
> > Many vendor's systems are designed around the assumption that "all tests
> > must pass" and, for the rare cases where tests don't pass, one is
> expected
> > to manually annotate the test as failing. This is problematic if you
> > suddenly import 10,000 tests for a feature that you haven't implemented
> > yet. Or even 100 tests of which 27 fail. I don't have a good solution for
> > this other than "don't design your test system like that" (which is
> rather
> > late). I presume the answer will look something like a means of
> > auto-marking tests as expected-fail on their first run after import.
>
> Afaik, this is what Mozilla does now with CSS ref tests. Fantasai, please
> correct me if I'm wrong.
>
> Tests are batch imported (monthly) and ran. New tests which fail are
> marked as such in a manifest file (that Mozilla hosts) and get skipped in
> test runs. Not sure what happens to existing tests which were previously
> passing and now fail, but they're probably either skipped too or
> investigated.
>
> Would be great to understand how WebKit (or would that be vendor
> specific?) and Microsoft plan to proceed with this.
>
>
All of the public / "upstreamed" WebKit ports basically work the way James
describes, except that the WebKit infrastructure has ways to mark entire
directory trees (or suites) of tests as expected to fail (or also should be
skipped).

At the moment, tests are imported sporadically, in a completely ad-hoc and
manual manner. There are people (including Rebecca) actively working to
change this :)

-- Dirk