Re: Requirements flags in reference files

Thanks Peter. This is all very helpful.

-R

From: Peter Linss <peter.linss@hp.com<mailto:peter.linss@hp.com>>
Date: Wednesday, June 26, 2013 6:25 PM
To: Rebecca Hauck <rhauck@adobe.com<mailto:rhauck@adobe.com>>
Cc: "\"Gérard Talbot\"" <css21testsuite@gtalbot.org<mailto:css21testsuite@gtalbot.org>>, Bem Jones-Bey <bjonesbe@adobe.com<mailto:bjonesbe@adobe.com>>, "public-css-testsuite@w3.org<mailto:public-css-testsuite@w3.org>" <public-css-testsuite@w3.org<mailto:public-css-testsuite@w3.org>>
Subject: Re: Requirements flags in reference files

It's perfectly valid for a reference file to have flags, and they should be present as needed, but they should only reflect requirements of the reference, not the test. e.g. if the test uses the ahem font, but the reference doesn't, then the reference should not have the 'ahem' flag.

The flags are used by the test harness and the build code.

The build code uses some flags like 'HTMLonly' and 'nonHTML' to control format conversions, and 'asis' to avoid re-serialization.

The harness will list conditions triggered by the flags, and tests can be filtered by flag (say, skip all the 'ahem' tests if you don't have it installed, or all the 'http' tests if you're running locally).

The harness also uses 'may' and 'should' to determine which tests are optional for computing CR exit requirements and generating implementation reports.

Other flags are for future use, like the 'animated' or 'interact' flags will tell an automatic reference comparator to not bother.

Also note that some flags will get injected automatically by the build code, such as 'reftest' when it detects reference links.

Peter

On Jun 26, 2013, at 4:45 PM, Rebecca Hauck wrote:

Hey Peter & Gérard,

I've seen Gérard remove these flags in the past [1], but the guidelines [2] are somewhat ambiguous. Should a ref file not have flags? For cases where the test and the reference have the same flag (for example, ahem), it makes sense that you needn't duplicate the info in the ref file as its bound to the test.  What about cases where the ref file has a flag that the test file does not? An example off the top of my head is maybe using SVG to render something equivalent that you're testing in CSS.

And more generally, how exactly are these flags used?  I saw some confusion about flags at TestTWF Tokyo as well that raises this question for me.

(cc'ing my colleague Bem, who's not subscribed to this list, but who's encountering this situation)

Thanks,
-Rebecca

[1] http://hg.csswg.org/test/rev/ed3f96bded71
[2] http://wiki.csswg.org/test/format#requirement-flags