W3C home > Mailing lists > Public > public-html-testsuite@w3.org > November 2010

Re: policy regarding testing open areas of the spec

From: James Graham <jgraham@opera.com>
Date: Thu, 25 Nov 2010 11:18:13 +0100
Message-ID: <4CEE37E5.5000403@opera.com>
CC: "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>
On 11/25/2010 10:50 AM, David Carlisle wrote:
> Following on slightly from James Graham's comment about metadata for tests,
> is there a policy for testing (or not) areas of the spec which are still
> subject to open bug reports (or other reasons for possible change)
> I have just pushed
> http://test.w3.org/html/tests/submission/DavidCarlisle/math-parse02.html
> Not to ask for formal approval (yet) but to get clarification on policy.
> This is testing innerHTML on math
> which is the subject of bug 11204, currently the file has the
> unstructured comment
> <!--
> http://www.w3.org/Bugs/Public/show_bug.cgi?id=11204
> -->
> But some structured way of denoting which part of the spec and/or
> resolution of issues a test depends on would be good I think.

I would say it is fine and, indeed, should be encouraged, to push tests 
for bugs. They can then be used to assess whether a spec change caused 
the bug to be fixed in the expected way, for example. However assuming 
our current process of manually approving tests; we should not allow 
such tests to be approved until the corresponding bug is fixed.

With metadata it would be nice to add an optional bug:<bug-id> field for 
such cases. I already think we should have a specref:[<identifier>] 
field to mark the particular assertions in the spec that a test tests. I 
think that having an annotated version of the full sepc showing test 
coverage, much like (and likely reusing infrastructure from) that Philip 
produced for the <canvas> tests [1] should be a short-term goal of the TF.

[1] http://philip.html5.org/tests/canvas/suite/tests/spec.html
Received on Thursday, 25 November 2010 10:18:54 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 15:49:37 UTC