Re: Support reference links in reftests on WebKit

On Aug 23, 2011, at 6:44 PM, Hayato Ito wrote:

> Hi, I am hayato, working on WebKit project.
> 
> I am planning to run reftests which are written in CSS2.1 test format,
> in our testing infrastructure of WebKit.
> https://bugs.webkit.org/show_bug.cgi?id=66295
> 
> I've found that a manifest file for reftests is now deprecated.
> Instead of the manifest file, using reference links seems to be
> encouraged in reftests.
> http://wiki.csswg.org/test/reftest
> http://wiki.csswg.org/test/css2.1/format (It seems that this page was
> updated on Aug 6).

Yes, I made those changes to the format and to the build system. It adds more capability by allowing 'OR' match relationships and the shift to links versus manifest files keeps the data bound to the tests so that it's less likely to get out of sync (which has happened in the past).

> 
> Therefore, we are going to support reference links in reftests. Is
> this a right direction to proceed?
> I appreciate you guys could give me any advise or suggestion if I am
> wrong or missing anything.

The build system still produces the manifest files and will continue to do so, they were only deprecated as input to the build, not from the output.

There's no harm in supporting the links in your testing system, but the only reason you'd need to is if you want to run tests without performing the build process.

Note that the more significant change here is the switch to reference chains and you should modify your testing system to support them either way. Specifically, if test 'A' has two 'match' references, 'B' and 'C', the old system would define:
== A B
== A C

whereas the new system defines this as:
== A B
== B C
== C B   (not entirely necessary, but there for completeness)

This means that test 'A' (or any test matching 'B' or 'C') must match BOTH 'B' and 'C'

The relationship is defined this way both in the reference links and the reference manifest build output.

In the new system if you have:
== A B
== A C

it means that 'A' may match EITHER 'B' or 'C', and further implies that 'B' and 'C' are not identical, which may (and probably should) be called out as an explicit mismatch between 'B' and 'C', i.e.:
== A B
== A C
!= B C
!= C B
or as links in the sources:
in 'A':
<link rel='match' href='B' />
<link rel='match' href='C' />

in 'B':
<link rel='mismatch' href='C' />

in 'C':
<link rel='mismatch' href='B' />

This allows much more flexible definitions of match/mismatch relationships. Note that the references may chain though multiple arbitrary levels and there will be cycles that need to be detected and resolved.

Peter

Received on Wednesday, 24 August 2011 17:33:48 UTC