Re: Support reference links in reftests on WebKit

On Aug 25, 2011, at 1:37 AM, Hayato Ito wrote:

> Hi Peter,
> Thank you for the explanation. That's really helpful for me to
> understand the current situation.
> 
> On Thu, Aug 25, 2011 at 2:32 AM, Linss, Peter <peter.linss@hp.com> wrote:
>> There's no harm in supporting the links in your testing system, but the only reason you'd need to is if you want to run tests without performing the build process.
> 
> Thank you. We don't have a plan to 'build' tests. So we are going to
> support only reftests which have reference links.

The build process isn't necessary, but on addition to the manifest files, it provides tests in alternate formats (i.e.: XHTML, HTML, and XHTML-Print) and index pages which you may find useful.

> 
>> 
>> Note that the more significant change here is the switch to reference chains and you should modify your testing system to support them either way. Specifically, if test 'A' has two 'match' references, 'B' and 'C', the old system would define:
>> == A B
>> == A C
>> 
>> whereas the new system defines this as:
>> == A B
>> == B C
>> == C B   (not entirely necessary, but there for completeness)
> 
> Yeah, it looks unnecessary to test B and C. I think there is an
> equivalence relation in all rendering results of test files.

To be clear, if you have:
test A with: <link rel='match' href='B' />

and reference B with: <link rel='match' href='C' />

then you absolutely _must_ match the test to both B and C. This is done for situations where reference 'B' relies on features that may in turn fail in such a way as to match a failing test. When that's a desk, you add another reference 'C' using yet another technique (which may also fail but in a different way).

The only part that isn't strictly necessary is then defining 'C' to match 'B', since the match is by definition reflexive. As a general practice, we do define the reflexive relationship in case there are other tests that link to 'C' directly, then they still pick up the match relationship to 'B'.

> 
>> or as links in the sources:
>> in 'A':
>> <link rel='match' href='B' />
>> <link rel='match' href='C' />
>> 
>> in 'B':
>> <link rel='mismatch' href='C' />
>> 
>> in 'C':
>> <link rel='mismatch' href='B' />
>> 
>> This allows much more flexible definitions of match/mismatch relationships. Note that the references may chain though multiple arbitrary levels and there will be cycles that need to be detected and resolved.
> 
> The last example is very useful for me to understand how reference
> links work in the new system. Thank you.
> 
> Let me ask one question. Does 'EITHER' apply to only 'match'? If a
> reftest test file has two 'mismatch' entries, does that mean 'AND'?
> 
> For example, does the following html mean ((A != B) AND (A != C))?
> 
> A:
>  <link ref='mismatch' href='B'>
>  <link ref='mismatch' href='C'>

That is currently the case yes. If you find (or can reasonably conceive of) a situation where you require an OR condition for mismatch, please let me know. That would mean that, in your example, 'A' could match 'B' so long as it doesn't match 'C' and vise versa. If that were the situation, you could simply define it as match 'B' or 'C' and then define 'B' != 'C'

Again, note that the relationships could get complicated, i.e.:
A == (B != C)
A == (D != C)

In this case, test 'A' may match either reference 'B' or reference 'C. Both reference 'B' and 'D' in turn must NOT match 'C'. With the ability to chain both 'match' and 'mismatch' the possible conditions can theoretically get unwieldy.

Peter

Received on Thursday, 25 August 2011 23:23:41 UTC