RE: Test Case Template/Meta Data

How about we agree on doing this...

Some test cases (the example we have been using) are too complex
to link back to a very specific part of the HTML5 spec.

Though we could at least categorize this test case as a complex
parser test and link it back to '9.2 Parsing HTML documents'.

We could then also be open to have someone write a specific test
that covers a specific part of section 9.2.*.*.

A template could be used with an xml file checked into CVS. So that
it would be possible to generate a test page similar to below using
script.

This would also enable test organization and ensure we are not
missing specific parts of the specification.  

<!DOCTYPE html>
<html>
<head>
  <title>9.2 Parsing HTML documents</title>
  <link rel="help" href="http://www.w3.org/TR/html5/syntax.html#parsing"/>
  <meta name="flags" content="complex" />
..
..
..
..
<iframe> || <object> tag could point to the test....

-Kris

-----Original Message-----
From: Geoffrey Sneddon [mailto:gsneddon@opera.com] 
Sent: Friday, November 20, 2009 7:07 AM
To: Kris Krueger
Cc: 'James Graham'; 'public-html-testsuite@w3.org'
Subject: Re: Test Case Template/Meta Data

Kris Krueger wrote:
>  * What parts of the HTML5 parsing does it specifically test? 9.2.2.4?

Given a test like:

<!doctype html PUBLIC "-//W3C//DTD HTML 3.2//en"><p><table>

Which expects a tree like:
| DOCTYPE html "-//W3C//DTD HTML 3.2//en" void
| html
|   head
|   body
|   p
|     table

What would you say this tests? It tests (off the top of my head, not 
actually running through this in the parser): 9.2.3, 9.2.4.1, 9.2.4.8, 
9.2.4.10, 9.2.4.45, 9.2.4.53, 9.2.4.54, 9.2.4.55, 9.2.4.56, 9.2.4.57, 
9.2.4.58, 9.2.4.59, 9.2.4.61, 9.2.5.1, 9.2.5.2, 9.2.5.4, 9.2.5.5, 
9.2.5.5, 9.2.5.6, 9.2.5.7, 9.2.5.9, 9.2.5.10, 9.2.5.12, 9.2.5.22, 
9.2.25, and 9.2.6.

Any parser test inevitably will test large parts of tokenization and 
tree-building as that is the only way to get to those states.

James' example of <table>x has the "x" being processed in 9.2.4.1, 
9.2.5.3, 9.2.5.10, 9.2.5.12, and 9.2.5.13: what does that eight 
character test test? If we can't quite easily agree upon where an eight 
character test should be classified, what hope do we have for more 
complex tests such as mine above?

I agree ideally we need to categorize the parser tests, but I'm not sure 
how easily they can be classified (I think the one classification we do 
need and is unambiguously needed is tests that are specific for 
scripting-enabled case and scripting-disabled case).

>  * What does failure look like?

The parsed DOM does not match the tree given. It should be obvious 
whether something passes or fails, I don't think we need to describe 
what failure looks like (heck, failure by nature can look like anything).

>  * What should we focus on adding next for parsing tests section 9.2.2.4, 9.2.5.26?

I think we can learn from the CSS WG that the difficulty is getting a 
test suite written at all, the difficulty is getting people writing 
tests. I think worrying about how we are going to know what section to 
write tests for is the least of our worries: we need to get tests 
written at all to start with, and for implementers at least, they will 
want tests for the entire parser (so caring about 9.2.2.4 or 9.2.5.26 is 
irrelevant), or for other parts of the spec, what they are implementing.

Trying to push people to write tests for a specific part of the spec 
seems futile unless we know we have people who can just write tests for 
any part of the spec.

-- 
Geoffrey Sneddon - Opera Software
<http://gsnedders.com/>
<http://www.opera.com/>

Received on Friday, 20 November 2009 17:27:00 UTC