W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2014

Re: Mismatch between CSS and web-platform-tests semantics for reftests

From: James Graham <james@hoppipolla.co.uk>
Date: Wed, 03 Sep 2014 14:41:16 +0100
Message-ID: <54071A7C.3080602@hoppipolla.co.uk>
To: public-test-infra@w3.org
On 20/08/14 01:22, Peter Linss wrote:

>> Are these features something that any actual implementation is 
>> running? As far as I can tell from the documentation, Mozilla 
>> reftests don't support this feature, and I guess from Dirke's 
>> response that Blink/WebKit reftests don't either. That doesn't 
>> cover all possible implementations of course.
> I'm actually in the middle of a big cleanup of our test harness and
> it will support this feature when I'm done (so far we haven't been
> able to represent the situation in our manifest files properly, I'm
> fixing that too).

So I was looking at adding this to web-platform-tests and the current
design adds some non-trivial complexity. As background,
web-platform-tests uses a script to auto-generate a manifest file with
the test files themselves being the only required input. This is
rather slow, since it involves actually parsing the *ML files and
inspecting their DOM. Therefore it is important to be able to perform
incremental updates.

Currently it is always possible to examine a single file and determine
what type of thing it represents (script test, reftest, manual test,
helper file, etc.). For example reftests are identified as files with
a <link rel=[mis]match> element. Since (unlike in CSS) tests in
general are not required to contain any extra metadata, allowing
references to link to other references introduces a problem because
determining whether a file is a reference or a test now requires
examining the entire chain, not just one file.

Obviously this isn't impossible to implement. It's just more
complicated than anything else in the manifest generation, all in
order to support a rarely-used feature. Are the benefits of the
approach where the data is distributed across many files really great
enough, compared to an alternate design where we put all the data
about the references in the test itself, to justify the extra
implementation burden? As far as I can tell the main benefit is that
if two tests share the same reference they get the same full chain of
references automatically rather than having to copy between files.
Received on Wednesday, 3 September 2014 13:41:44 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:11 UTC