Re: Mismatch between CSS and web-platform-tests semantics for reftests

On 04/09/14 18:37, Peter Linss wrote:

> We do also keep our tests ready to run "as-is" directly from the
> repo, the build step does provide the format conversions so that all
> tests are available in both HTML and XHTML (except where it doesn't
> make sense).

Sorry if I'm being dozy but I don't really understand what the upshot of
this is. Does it mean that the tests are all runnable directly from the
repo, but you might not get the same test in both XHTML and HTML unless
you run the build? Or are there some tests that aren't directly runnable?

>> So your approach is to regard each file with one or more
>> mis[match] links as a test and then to allow the overall result of
>> one test to depend on other tests? I guess that works.
> 
> No. Any file with a <link rel='help' /> is a test (note that what we
> really care about is that the test is related to a spec through some
> kind of metadata, not the rel='help' specifically, it'd be a trivial
> change to infer the link based on the files path in the repo). If a
> test _also_ has a [mis]match link, then it's a reference test. If the
> test contains a <script src='/resources/testharness.js'> then it's a
> script test.

OK, so we don't enforce the presence of rel=help links, so we can't use
that to identify tests. But I wonder if just basing things on the
presence of rel=[mis]match is good enough. If it is that seems like a
way to get approximate semantic compatibility between CSS and wpt. Is it
possible to have a situation where it will give bogus results e.g. if A
must match B or C and C must match D, can there be a legitimate test in
which A matches B and the C and D failing to match is misleading (this
would be true for D == A, for example, but loops have to be treated
specially anyway I guess).

> The naming convention is used by the build tools (and Shepherd) to
> differentiate between a test, support file and a reference. Given
> that tests and support files can also be used as references it's more
> of a convenience. In built test suites all reference file get put
> into a 'reference' directory regardless of their location in the
> source tree.
> 
> Tests do have to meet the test naming convention, which is basically:
> don't be a reference file, don't be a support file (i.e. in a
> 'support' directory), and don't be a tool file (i.e. in a 'tools'
> directory), and don't be in a directory that is ignored.

OK.

>>> Also, let me point out again, that the bulk of the code we use
>>> for our build process is in a factored out python library[1]. It
>>> can use some cleanup, but it contains code that parses all the
>>> acceptable input files and extracts (and in some cases allows
>>> manipulation of) the metadata. Shepherd also uses this library to
>>> do it's metadata extraction and validation checks. If we can
>>> converge on using this library (or something that grows out of
>>> it) then we don't have to re-write code managing metadata... I'm
>>> happy to put in some effort cleaning up and refactoring this
>>> library to make it more useful to you.
>> 
>> Can you point to the exact code that you have in mind? Lots of that
>> code seems to be concerned with problems that I'm not trying to
>> solve at this time; literally all I want is to be able to get a
>> complete set of test files and the data required to run the tests
>> automatically, in a way that works for w-p-t and CSS without
>> special casing.
> 
> The code that should be interesting to you would be mostly in
> Sources.py[1]
> 
> The SourceTree class will, given a file path, determine if the file
> is a test, reference, support, or tool file. Its currently hard-coded
> against the css repo layout but will likely be mostly compatible with
> the wpt repo at this point.
> 
> The SourceCache class will (via the generateSource method) provide an
> instance of a FileSource subclass (and use a cache, obviously). You
> can then query the FileSource for its metadata.
> 
> These classes can be used pretty much by themselves. And as I said
> before, a lot of that code is old and crufty and can use some TLC
> (much of it dates back 7 or 8 years). I'm happy to do some cleanups
> and refactoring for you if you let me know your specific needs.

OK, I'll take a look. Thanks.

Received on Sunday, 7 September 2014 15:04:31 UTC