Re: Coverage analysis

On Tuesday, February 26, 2013 at 8:31 PM, Rebecca Hauck wrote:
> I generated some reports for a couple of CSS specs (Transforms and
> Backgrounds & Borders) and used some of the CSS infrastructure. I still
> used Robin's spec parser, but pulled the number of tests from our existing
> DB (from the annotations on the specs).
> 
> A few changes to Robin's scripts to support this:
> 
> - (minor) Parse the ToC whether it's an ol or a ul - CSS specs use the
> latter; HTML the former
> - Bypass the test-per-section.json and pull the test counts directly from
> the spec if it has these annotations.
> 
> Both methods can peacefully coexist now, but we may want to discuss
> converging them (like if the other Wgs wanted to adopt annotate.js).
> 
> I sent a pull request with these changes:
> https://github.com/w3c/html-testsuite/pull/33
> 
> In this pull request, I've added a separate css-index.html for the CSS
> reports with the accompanying json files. I realize the html-testsuite
> repo is probably not the logical home for these, but perhaps we can
> discuss a more centralized place for all things coverage in the Testing
> Task Force.

Agreed. The goal is to rename the html-testsuite repository and include all OWP test suites in it.
> I had trouble building gh-pages on my fork of the entire html-testsuite,
> so I temporarily have the CSS reports here until the pull request is
> merged or they're moved elsewhere:
> 
> http://rhauck.github.com/css-coverage/
> 
> There are a few flaws in these reports (for example, the Backgrounds &
> Borders report probably doesn't need to include changes, acknowledgements,
> and indices). I'll write up a list of ideas and things I noticed as I dug
> deeper into this. Again, perhaps a good thing to address in the Task Force.

Sounds like we should be improving our heuristics for CSS to include all the propdef blocks. 
> 
> Robin, thanks again for getting this going! It was well documented and
> straightforward to adapt, which is always appreciated. Now that we have
> something functioning, it should be easy tweak & refine it to make this
> more useful over time. I'm happy to help move it forward!

Great work on this. Thanks for sharing.

Worked on the UI a bit to offer a more opinionated view which is thus much easier to parse:

http://tobie.github.com/css-coverage/index.html

I'm planning to generalize and automate this process for all specs we'll be testing in the near future.

--tobie

Received on Wednesday, 6 March 2013 18:05:34 UTC