Re: Towards a better testsuite

> On Mar 29, 2016, at 05:26, Geoffrey Sneddon <me@gsnedders.com> wrote:
> 
> On 28/03/16 04:47, Florian Rivoal wrote:
>> As far as the path of the file carying information about the spec it is
>> testing, that's sort of true, but it doesn't indicate which part of a
>> spec, which is useful, and it doesn't help when testing multiple specs
>> at a time.
> 
> In wpt the path does indicate what section of the spec it is testing:
> there's far more levels of directories there than there are in
> csswg-test, and I think that's a good thing, as it makes it far easier
> to find tests.

Using URLs seem to me to be a more reliable way to indicate a particular section of a spec than storing the test case in a specially named folder that implies the same thing, but as long as the info is there, that's a matter of preference more than objections.

It seems much easier to build the linter/commit hook to verify that the URL exists than to build it to check that the section implied by the path exists (what's the convention for converting the html outline or anchors to a path?), though. If we don't adopt strict conventions in path names, it seems it would not work, and if we do, I am not sure we're saving any overhead compared to having URLs in the meta data.

And the path approach still doesn't allow to point to multiple specs or multiple parts of a spec (unless you turn the file tree into a file graph using symlinks, but I don't think anybody wants to go there).

>>> For wpt we have master on w3c-test.org <http://w3c-test.org/>, though
>>> that doesn't quite suffice for running them all given some rely on
>>> specific host names resolving. It's also not the easiest way to run
>>> reftests, but I'm not sure what can be done to make that easier:
>>> determining pixel by pixel equivalence by eye will always be hard (and
>>> really, quickly changing between tabs/frames can easily end up with
>>> things being incorrectly marked, it only really makes sense to do
>>> programmatically).
>>> 
>> Having it on the master branch mean you can't use it for reviews.
> 
> PRs from a whitelisted set of users get automatically mirrored to
> http://w3c-test.org/submissions/ and PRs from other users can be added
> to that by any of the whitelisted users (there's arbitrary code
> execution going on server-side, so we can't just throw up any PR).

CSS tests rarely (never?) have a server side component, so maybe we're
in a better situation from a security standpoint.

But yes, something along these lines (preferably with some kind of
affordance for ref tests like on shepherd) would be great.

 - Florian

Received on Tuesday, 29 March 2016 00:33:57 UTC