Re: Testing Requirements issues - some proposed solutions

On Thursday, March 8, 2012, 10:02:18 PM, Peter wrote:

LP> On Mar 8, 2012, at 11:51 AM, Tavmjong Bah wrote:
>> On Thu, 2012-03-08 at 20:24 +0100, Chris Lilley wrote:

>>> Similarly we should have a license statement, which the original SVG tests had and the later ones dropped, again it can be mostly a link.

LP> I like the link idea, I'm going to suggest that the CSS WG adopt
LP> that too (and add some support for it in Shepherd).

We now have

<link rel="license"
         href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license.html"/>


>> Lets make it a link to keep the files compact.

yes, we agreed.


LP> FWIW, the CSS test suite build system captures the Mercurial
LP> revision information in the manifest file. During it's import, the
LP> test framework has the capability to run a normalized diff of
LP> tests who's revision information has changed (to detect changes
LP> only in non-functional metadata). This is currently not exposed in
LP> the W3C's instance admin interface. If the import see different
LP> revision ids, but no changes outside some of the metadata, it considers the tests identical.

That sounds a lot better.

>>> *    CSS has a meta section for "flags" to indicate what features are needed by the renderer for the test. Two flags relevant to SVG are: "svg" and "namespaces". Since SVG is SVG, do we need to include these flags or can we just assume that any SVG renderer will support SVG and namespaces? 

>>> We need the flags because they are used by the build system and because tests can be shared between groups (for example the CSS WG can use tests that we make, for specs that are jointly developed). Similarly the CSS WG makes XHTML tests and they have the xhtml flag despite the fact that all CSS/HTML renderers will understand html. Which helps us, since those tests can not be sent to SVG-only renderers.

LP> Sort of. The build system uses some of the flags (as does the
LP> test framework), others are merely reported to the user running
LP> the test. It's not strictly necessary to flag all SVG tests as
LP> 'svg'. For example, in the CSS suites, we presume all tests can be
LP> re-generated in HTML and XHTML, only using the 'HTMLOnly' and
LP> 'nonHTML' flags as needed. This part is configurable.

Right, but the presumption that all tests can be regenerated in html and xhtml is incorrect :)


LP> Where tests apply to multiple specs (and contain links to non SVG
LP> specs), then a 'sag' flag is more useful as the test will show up
LP> in test suites for the other specs. This is your call.

Yes, it seems better if we flag svg tests explicitly.

LP> The set of allowable flags is open ended, define any you need
LP> (but no more than you really need). You can tell the test
LP> framework on a spec by spec basis which flags mean optional tests.
LP> So if a test is for multiple specs, and tests optional features of
LP> one, but not all, make up a flag that means that (e.g. if a
LP> feature is 'should' or 'may' in CSS but 'must' in SVG, the test needs a flag like 'mayCSS').

OK.

>>> It is not clear to me if a multi namespace XHTML + SVG (or html5 + svg) test can simply indicate that with flags="xhtml svg" or if we need a new "html+svg" flag as well.


LP> You only need flags that by presence or absence determine build
LP> options, or framework features. So if you have a XHTML+SVG source
LP> file, and want the build system to output HTML+SVG and XHTML+SVG,
LP> you need flags that define the combinations of possible output. We
LP> can set the default to anything you want on a per-sepc basis, so
LP> the default could be: no flag == XHTML+SVG and HTML+SVG, then
LP> re-use the 'HTMLOnly' and 'non'HTML' flags to control your build options.

OK

>>> Not listed as an issue, but it is an issue:

>>> * We previously had per-test description, instructions for running the test (weirdly named 'operator script') and explicit pass criteria. Those were useful, especially the pass criteria. 

>>> In the old system, our build script extracted that and put it into the test harness so it would be displayed along with the actual SVG test and the reference image.

>>> Where do those go, in the new system? And how do they get displayed, since the test harness is auto generated and won't pull in that information?

>> They go in the <meta name="assert"  ... /> section.

LP> Note you can have multiple <meta name="assert" /> tags, the framework will display them all.

We noticed on todays call that we can't use meta, because if a meta occurs in an svg context, the html5 parser will auto close the svg and drop into html.

We can use link, however.

So we modified our proposed template to use the svg metadata element to put the flags in, and the desc element to put the assertions in.

   <metadata class="flags">TOKENS" </metadata>
   <desc class="assert">TEST ASSERTION</desc>

Will that be ok for the system to detect and import?

>>> * we need a similar, but simpler, template for the reference images.

>> CSS uses an identical format except without <link>s and <meta>s (except
>> that the author <link> is kept).

LP> The CSS reference files also use the <link rel='match' /> and
LP> <link rel='mismatch' /> elements to build reference chains (where
LP> all must match vs one must match). The only metadata forbidden in reference files is spec links.

LP> The actual rules are:
LP> * Tests: MUST have Title, spec links, and author. SHOULD have
LP> assert. Approved tests SHOULD have reviewer. Other metadata optional.
LP> * Reference files MUST have author, MUST NOT have spec links.
LP> Approved reference files  SHOULD have reviewer. Other metadata optional.
LP> * Support files (.css, .png, etc) all metadata optional.

Thanks for the clarifications.




-- 
 Chris Lilley   Technical Director, Interaction Domain                 
 W3C Graphics Activity Lead, Fonts Activity Lead
 Co-Chair, W3C Hypertext CG
 Member, CSS, WebFonts, SVG Working Groups

Received on Thursday, 8 March 2012 21:31:07 UTC