Re: Towards a better testsuite

On 29/03/16 01:33, Florian Rivoal wrote:
> 
>> On Mar 29, 2016, at 05:26, Geoffrey Sneddon <me@gsnedders.com> wrote:
>>
>> On 28/03/16 04:47, Florian Rivoal wrote:
>>> As far as the path of the file carying information about the spec it is
>>> testing, that's sort of true, but it doesn't indicate which part of a
>>> spec, which is useful, and it doesn't help when testing multiple specs
>>> at a time.
>>
>> In wpt the path does indicate what section of the spec it is testing:
>> there's far more levels of directories there than there are in
>> csswg-test, and I think that's a good thing, as it makes it far easier
>> to find tests.
> 
> Using URLs seem to me to be a more reliable way to indicate a particular section of a spec than storing the test case in a specially named folder that implies the same thing, but as long as the info is there, that's a matter of preference more than objections.
> 
> It seems much easier to build the linter/commit hook to verify that the URL exists than to build it to check that the section implied by the path exists (what's the convention for converting the html outline or anchors to a path?), though. If we don't adopt strict conventions in path names, it seems it would not work, and if we do, I am not sure we're saving any overhead compared to having URLs in the meta data.
> 
> And the path approach still doesn't allow to point to multiple specs or multiple parts of a spec (unless you turn the file tree into a file graph using symlinks, but I don't think anybody wants to go there).

But graph-like structures are fun! :)

There's certainly some extra complexity here from the versioning of the
testsuites. (FWIW, thus far we've avoided any need for versioning for
wpt, which seems to have worked fine.) Going forward, we essentially
have module testsuites which are supersets of the 2.1 testsuite which is
a superset of the 1 testsuite. (Most tests that point to multiple specs
are simply pointing to 2.1 and 3 definitions of the same feature.)

That said, we haven't exactly maintained the level 1 testsuite, so maybe
we can follow a wpt-like approach and only maintain the latest version.
Maybe that's too radical, though. :)

>>>> For wpt we have master on w3c-test.org <http://w3c-test.org/>, though
>>>> that doesn't quite suffice for running them all given some rely on
>>>> specific host names resolving. It's also not the easiest way to run
>>>> reftests, but I'm not sure what can be done to make that easier:
>>>> determining pixel by pixel equivalence by eye will always be hard (and
>>>> really, quickly changing between tabs/frames can easily end up with
>>>> things being incorrectly marked, it only really makes sense to do
>>>> programmatically).
>>>>
>>> Having it on the master branch mean you can't use it for reviews.
>>
>> PRs from a whitelisted set of users get automatically mirrored to
>> http://w3c-test.org/submissions/ and PRs from other users can be added
>> to that by any of the whitelisted users (there's arbitrary code
>> execution going on server-side, so we can't just throw up any PR).
> 
> CSS tests rarely (never?) have a server side component, so maybe we're
> in a better situation from a security standpoint.
> 
> But yes, something along these lines (preferably with some kind of
> affordance for ref tests like on shepherd) would be great.

We currently rely upon Apache's .htaccess files, which dependent on the
AllowOverride setting could potentially lead to security issues; we
don't currently actually have any server-side scripts in csswg-test, we
just rely on hixie.ch for that… (and that's something we badly need to
change!).

/Geoffrey

Received on Tuesday, 29 March 2016 07:49:04 UTC