Re: Towards a better testsuite

> On Mar 25, 2016, at 02:00, Geoffrey Sneddon <me@gsnedders.com> wrote:
> 
> The current status, as I understand it, is: test262 I believe people
> are mostly running old versions of and contributing little to;
> Microsoft is running weekly updated versions of csswg-test and Gecko
> is running a several-year-old version with no realistic plan to update
> it, nobody contributes that much (a tiny subset of Gecko stuff is
> automatically synced, but the vast majority is not);
> web-platform-tests is run by Microsoft semi-regularly, is run with
> two-way syncing from Gecko and Servo, with plans by Blink and
> Microsoft to get there AIUI, and with more in the way of contributions
> than either of the other two repositories. WebKit just aren't running
> anything, far as I'm aware. The only other group I'm aware of running
> anything is Prince, running a small subset of an old version of
> csswg-test.

We run (a growing subset of) csswg-test at Vivliostyle as well.

> That said, I think it's worthwhile to reiterate that requiring *any*
> metadata causes friction. Tests written by browser vendors are rarely
> a file or two which is quick to add metadata too. I know in general
> people seem interested in using the same infrastructure to run both
> web-platform-tests and csswg-test, which essentially requires the
> metadata required to run the tests be identical across the two.

I think you need to be very careful. Yes, removing all meta data
lowers friction for test submission, but it increases it
when you're on the receiving end.

Presumably, when someone adds a test to a browser's private repo,
even if there's no explicit metadata at all in the test, there
is implicit knowledge about what this is about. Maybe in a bug
tracker somewhere. Maybe based on the branch its being added to.
Maybe just because the person who added it is or knows the person
who's supposed to make it pass.

But this contextual information isn't passed along when sharing the
test with other vendors. If some other vendor syncs with a repo
from someone else with tests in it that have 0 meta data, regularly
they'll wake up to a test run with a few hundred failing tests,
and no indication whatsoever what these tests are about. Depending on
the test, finding out can be far from obvious. This is a great way
to make sure that failing tests are ignored, or that we stop syncing.

Not sure if that's better or worse, but if you have in that
lot incorrect tests that pass even though they should really
not, you'll wind up integrating them in your regression test suite
(hey, these tests used to pass, we need to keep them green)
without even being aware of it.

So I'm in favor of as little meta-data as possible, but not of
no meta data at all. As a consumer of tests, the assertion and
the link to the related specs are very important.


> The other notable difference is in tooling: Mercurial is used with a
> git mirror, and then reviews are done split across Shepherd and
> public-css-testsuite and some issues filed on GitHub, and with some
> people expecting some pre-landing review through GitHub PRs and with
> some people pushing directly to Mercurial… Really everything would be
> simpler if we had *one* single way to do things. I'd much rather have
> everything on GitHub, review happening on PR submission, and nits and
> such like be reported as GitHub issues. This keeps everything in one
> place with tools most people are used to.

One system is better than two, and my personal preference goes to GitHub
as well, although I can deal with the alternative.

One of the problem with the GitHub workflow is that it doesn't include
an easy way to run/preview the tests. Sure, you can always check it out
locally and run the tests there, but that's quite a bit of overhead
compared to the previews offered by shepherd.

If we move to a github based workflow (which I would support), I
would want a replacement of some kind to that.

> To outline what I'd like to see happen:
> 
> - Get rid of the build system, replacing many of it's old errors with
> a lint tool that tests for them.

Fine by me.

> - Policy changes to get rid of all metadata in the common case.

Reduce, yes. Get rid of all, no.

I don't care about the title. It's nice if it's well though out, but it's not that important.

Authors and reviewers (at least for my purposes) are adequately covered by git/hg.

I want the assertion and the spec links (at the very least one of the two, but I really want both).

For flags, as you explored in your earlier mail, we can get rid of most. But some (which aren't needed for most tests, but significant for some) should stay:
- animated
- interact
- may
- should
- paged
- scroll
- userstyle

If no flag applies to your test, you should be allowed to omit the meta entirely. Rejecting a test because it has no flags meta when it should have "<meta name='flags' content='' />" doesn't seem worth the overhead.

> - Change the commit policy. (Require review first, require no new lint errors.)

Fine by me.

> Long-term it's probably worth considering merging the whole thing into
> web-platform-tests, so we have all the W3C tests in one place.

I don't care strongly either way.

 - Florian

Received on Friday, 25 March 2016 04:39:05 UTC