Re: Thinking about test coverage

It's not just about F&O, of course...

I've cross-linked many of the entries in changes.xml to the PR number, and going forward I suggest we aim to use the covers40 attribute to point directly to the PR rather than to an entry in changes.xml. We might eventually then take changes.xml out of the picture entirely, and just marry the GitHub tags with the covers40 attributes,

One of the problems of course is that there may be one or two tests for a feature, but not enough to consider it tested. For example invisible-xml has a token 3 or 4 tests at the moment just so you get an error if it's not implemented at all.

We've probably not been removing the "tests needed" labels in Github when we consider a feature sufficiently tested.

And of course there's the XSLT test suite as well.

Mike

> On 13 Mar 2024, at 16:48, Norm Tovey-Walsh <norm@saxonica.com> wrote:
> 
> Hi folks,
> 
> At yesterday’s meeting, I said I’d try to help improve the state of the test suite wrt what’s in, what’s out, what’s missing, and what’s needed.
> 
> I decided to see if I could build a view of the current state, as a precursor to working out what needs to be improved. To that end, see:
> 
>   https://qt4cg.org/test-coverage/
> 
> It’s very rough, but it shows:
> 
> 1. A summary of the PRs still marked “tests needed”.
> 
> 2. Test coverage, generated by comparing an index of the functions in F&O with the @covers-40 attributes in the test sets and test cases.
> 
> 3. Test anti-coverage, that is, @covers-40 attributes that seem to identify functions that are not currently in the F&O specification.
> 
> 4. A summary of all the tests showing the ‘spec’ and ‘feature’ dependencies.
> 
> It doesn’t feel exactly right, but it’s taken more time than I intended today. I’ll come back to it again tomorrow or over the weekend perhaps. It would be possible to automate building this page, but I haven’t attempted to do that (and I constructed a couple of ad-hoc indexes to produce it.)
> 
> One thing I’m still trying to work out is how we can leverage cross referencing to improve the view. For example, the PR 823 is still listed as “Tests Needed” but the “array-sort” test set is marked as related to PR 823. Does that mean the tests are written? Does it *always* mean that?
> 
> I think there are parallel questions about what markup we can use to track this better and how much of the tracking will (have to) remain a manual process.
> 
> Thoughts, etc. most welcome.
> 
>                                        Be seeing you,
>                                          norm
> 
> --
> Norm Tovey-Walsh
> Saxonica

Received on Wednesday, 13 March 2024 17:36:04 UTC