Thinking about test coverage

Hi folks,

At yesterday’s meeting, I said I’d try to help improve the state of the test suite wrt what’s in, what’s out, what’s missing, and what’s needed.

I decided to see if I could build a view of the current state, as a precursor to working out what needs to be improved. To that end, see:

   https://qt4cg.org/test-coverage/

It’s very rough, but it shows:

1. A summary of the PRs still marked “tests needed”.

2. Test coverage, generated by comparing an index of the functions in F&O with the @covers-40 attributes in the test sets and test cases.

3. Test anti-coverage, that is, @covers-40 attributes that seem to identify functions that are not currently in the F&O specification.

4. A summary of all the tests showing the ‘spec’ and ‘feature’ dependencies.

It doesn’t feel exactly right, but it’s taken more time than I intended today. I’ll come back to it again tomorrow or over the weekend perhaps. It would be possible to automate building this page, but I haven’t attempted to do that (and I constructed a couple of ad-hoc indexes to produce it.)

One thing I’m still trying to work out is how we can leverage cross referencing to improve the view. For example, the PR 823 is still listed as “Tests Needed” but the “array-sort” test set is marked as related to PR 823. Does that mean the tests are written? Does it *always* mean that?

I think there are parallel questions about what markup we can use to track this better and how much of the tracking will (have to) remain a manual process.

Thoughts, etc. most welcome.

                                        Be seeing you,
                                          norm

--
Norm Tovey-Walsh
Saxonica