Re: policy regarding testing open areas of the spec

On 11/25/2010 10:50 AM, David Carlisle wrote:
>
>
> Following on slightly from James Graham's comment about metadata for tests,
> is there a policy for testing (or not) areas of the spec which are still
> subject to open bug reports (or other reasons for possible change)
>
> I have just pushed
>
>
> http://test.w3.org/html/tests/submission/DavidCarlisle/math-parse02.html
>
> Not to ask for formal approval (yet) but to get clarification on policy.
> This is testing innerHTML on math
> which is the subject of bug 11204, currently the file has the
> unstructured comment
>
> <!--
> THIS TEST ASSUMES THAT
> http://www.w3.org/Bugs/Public/show_bug.cgi?id=11204
> IS RESOLVED
> -->
>
> But some structured way of denoting which part of the spec and/or
> resolution of issues a test depends on would be good I think.

I would say it is fine and, indeed, should be encouraged, to push tests 
for bugs. They can then be used to assess whether a spec change caused 
the bug to be fixed in the expected way, for example. However assuming 
our current process of manually approving tests; we should not allow 
such tests to be approved until the corresponding bug is fixed.

With metadata it would be nice to add an optional bug:<bug-id> field for 
such cases. I already think we should have a specref:[<identifier>] 
field to mark the particular assertions in the spec that a test tests. I 
think that having an annotated version of the full sepc showing test 
coverage, much like (and likely reusing infrastructure from) that Philip 
produced for the <canvas> tests [1] should be a short-term goal of the TF.

[1] http://philip.html5.org/tests/canvas/suite/tests/spec.html

Received on Thursday, 25 November 2010 10:18:54 UTC