Re: Testing Requirements issues - some proposed solutions

Hi Peter,

 The SVG WG just resolved on the format for our tests. You can see the
format at the bottom of:

http://www.w3.org/Graphics/SVG/WG/wiki/SVG2/Testing_Requirements

Your comments would be appreciated.

Note: In SVG in HTML, <link> won't break out of SVG mode but <meta> will
thus we've chosen to use <desc> for "TEST ASSERTION" and <metadata> for
"TOKENS".

     Tav
 

On Thu, 2012-03-08 at 21:02 +0000, Linss, Peter wrote:
> On Mar 8, 2012, at 11:51 AM, Tavmjong Bah wrote:
> 
> > On Thu, 2012-03-08 at 20:24 +0100, Chris Lilley wrote:
> >> Hello Tav,
> >> 
> >> Looking at the issues towards the end of
> >> http://www.w3.org/Graphics/SVG/WG/wiki/SVG2/Testing_Requirements
> >> 
> >> I have some suggestions for them.
> >> 
> >> *   We will need a new NS URL for SVGTestCase. 
> >> 
> >> Yes. The existing one is
> >> http://www.w3.org/2000/02/svg/testsuite/description/
> >> which doesn't point to anything (nor need it point to anything, to be clear).
> >> 
> >> The reason for using another namespace is to follow our own rule about putting non-SVG things in a different namespace.
> >> 
> >> Since it doesn't much matter I propose
> >> http://www.w3.org/2012/svgtest/
> > 
> > Cameron's proposal negates the need for this (not using HTML <head>). We
> > should discuss his proposal.
> > 
> >> *   Do we need to keep the copyright section? The CSS tests do not include a copyright. 
> >> 
> >> Yes we should have a copyright statement but it can be a link, it need not be a whole paragraph of text.
> >> 
> >> Similarly we should have a license statement, which the original SVG tests had and the later ones dropped, again it can be mostly a link.
> 
> I like the link idea, I'm going to suggest that the CSS WG adopt that too (and add some support for it in Shepherd).
> 
> >> 
> >> Here is some info that describes the test suite licenses
> >> http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html
> >> 
> >> Their suggested markup (needs an enclosing element, such as <p> in the html namespace)
> >> 
> >> Distributed under both the <a href="http://www.w3.org/Consortium/Legal/2008/04-testsuite-license">W3C Test Suite License</a> and the <a href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license">W3C 3-clause BSD License</a>. To contribute to a W3C Test Suite, see the <a href="http://www.w3.org/2004/10/27-testcases">policies and contribution forms</a>.
> >> 
> >> We could either include all that, or put it in a page somewhere and point to it.
> > 
> > Lets make it a link to keep the files compact.
> > 
> >> *    The CSS that the CSS test is testing is included in the <head> section where it can easily be found. This idea would be nice for SVG tests that test CSS so the CSS can be displayed along with the test (example). 
> >> 
> >> Yes. Originally we were writing the tests in one format then stripping out the non-SVG namespace stuff (mainly so Batik didn't complain about DTD validity, as I recall). 
> >> 
> >> * However we cannot put our CSS in the same place. We need a style section outside the SVGTestCase block with an appropriate 'id'.
> >> 
> >> Do we? Why? 
> > 
> > In CSS it is in the <head> section which in our case would be hidden by
> > the "d" namespace.
> >> 
> >> *   Do we need to keep the revision displayed in the SVG or can we rely on Mecurial? 
> >> 
> >> I suggest not.
> > 
> > Good!
> > 
> >> We originally added it because we sometimes found that a test had changed while a patch had not, so we wanted a visual check that they were the same revision.
> >> 
> >> Then we found that often, a test would be changed (for example to clarify the description) but the patch was still correct, and needed to be updated. Then this happened so often that we just ignored the in-your-face revision number.
> >> 
> >> So I suggest we don't display it.
> 
> FWIW, the CSS test suite build system captures the Mercurial revision information in the manifest file. During it's import, the test framework has the capability to run a normalized diff of tests who's revision information has changed (to detect changes only in non-functional metadata). This is currently not exposed in the W3C's instance admin interface. If the import see different revision ids, but no changes outside some of the metadata, it considers the tests identical.
> 
> >> 
> >> *    Do we need the "Not Approved" warning box? The adopted test directory structure keeps the approved tests separate from unapproved tests. 
> >> 
> >> And thus, we don't need it because the risk of going through a series of tests and accidentally seeing an unapproved one is gone.
> > 
> > Good!
> > 
> >> *    CSS has a meta section for "flags" to indicate what features are needed by the renderer for the test. Two flags relevant to SVG are: "svg" and "namespaces". Since SVG is SVG, do we need to include these flags or can we just assume that any SVG renderer will support SVG and namespaces? 
> >> 
> >> We need the flags because they are used by the build system and because tests can be shared between groups (for example the CSS WG can use tests that we make, for specs that are jointly developed). Similarly the CSS WG makes XHTML tests and they have the xhtml flag despite the fact that all CSS/HTML renderers will understand html. Which helps us, since those tests can not be sent to SVG-only renderers.
> 
> Sort of. The build system uses some of the flags (as does the test framework), others are merely reported to the user running the test. It's not strictly necessary to flag all SVG tests as 'svg'. For example, in the CSS suites, we presume all tests can be re-generated in HTML and XHTML, only using the 'HTMLOnly' and 'nonHTML' flags as needed. This part is configurable.
> 
> Where tests apply to multiple specs (and contain links to non SVG specs), then a 'sag' flag is more useful as the test will show up in test suites for the other specs. This is your call.
> 
> >> 
> >> We may also need additional flags 
> >> 
> >> - font if the test requires a specific font to be installed to run the test
> >> - optional if the test is for an optional feature
> >> 
> >> If we have multiple conformance classes then we may need flags for those. For example, fully color managed user agents vs. minmally color managed on unmanaged ones.
> 
> The set of allowable flags is open ended, define any you need (but no more than you really need). You can tell the test framework on a spec by spec basis which flags mean optional tests. So if a test is for multiple specs, and tests optional features of one, but not all, make up a flag that means that (e.g. if a feature is 'should' or 'may' in CSS but 'must' in SVG, the test needs a flag like 'mayCSS').
> 
> >> 
> >> It is not clear to me if a multi namespace XHTML + SVG (or html5 + svg) test can simply indicate that with flags="xhtml svg" or if we need a new "html+svg" flag as well.
> > 
> > OK, we'll keep.
> 
> You only need flags that by presence or absence determine build options, or framework features. So if you have a XHTML+SVG source file, and want the build system to output HTML+SVG and XHTML+SVG, you need flags that define the combinations of possible output. We can set the default to anything you want on a per-sepc basis, so the default could be: no flag == XHTML+SVG and HTML+SVG, then re-use the 'HTMLOnly' and 'non'HTML' flags to control your build options. 
> 
> > 
> >> Not listed as an issue, but it is an issue:
> >> 
> >> * We previously had per-test description, instructions for running the test (weirdly named 'operator script') and explicit pass criteria. Those were useful, especially the pass criteria. 
> >> 
> >> In the old system, our build script extracted that and put it into the test harness so it would be displayed along with the actual SVG test and the reference image.
> >> 
> >> Where do those go, in the new system? And how do they get displayed, since the test harness is auto generated and won't pull in that information?
> > 
> > They go in the <meta name="assert"  ... /> section.
> 
> Note you can have multiple <meta name="assert" /> tags, the framework will display them all.
> 
> > 
> >> Also
> >> 
> >> * we need a similar, but simpler, template for the reference images.
> > 
> > CSS uses an identical format except without <link>s and <meta>s (except
> > that the author <link> is kept).
> 
> The CSS reference files also use the <link rel='match' /> and <link rel='mismatch' /> elements to build reference chains (where all must match vs one must match). The only metadata forbidden in reference files is spec links.
> 
> The actual rules are:
> * Tests: MUST have Title, spec links, and author. SHOULD have assert. Approved tests SHOULD have reviewer. Other metadata optional.
> * Reference files MUST have author, MUST NOT have spec links. Approved reference files  SHOULD have reviewer. Other metadata optional.
> * Support files (.css, .png, etc) all metadata optional.
> 
> 
> Peter
>

Received on Thursday, 8 March 2012 21:25:07 UTC