some gaps in our tests (quick and dirty coverage check)

For what it's worth, I have been working on tools to perform a simple
measurement of the coverage of our test set.  The short version:

  1 There are (predictably) no occurrences of the nonterminals 'version'
    and 'prolog', or the choice that involves them, in any of the
    grammars of the test set.

  2 There are no hexadecimal numbers with length > 2.

  3 There is only one empty set (an empty inclusion); there are no empty
    exclusions.

I hope to develop a pull request to fill these gaps as soon as I can; if
anyone can beat me to the punch, please do so!

So far, I have run my coverage tools only on the spec grammar, since it
is used more than any other in the test set, and I have tested only
'choice coverage' in positive test cases; for a given grammar, the tools
check to see

  (a) whether every nonterminal in that grammar appears in the raw parse
      tree for at least one positive test case, and
      
  (b) whether every branch on every choice in the grammar is taken in at
      least one parse tree; for purposes of these tools, an option (x?)
      is a choice between () and (x), (x*) is a choice between (), (x),
      and (x, x+), and (x+) is a choice between (x), (x, x), and (x, x,
      x+).

If there are zero counts for either of these, there is a gap in the test
set's coverage for the relevant constructs.  As can be seen, there are
relatively few gaps in our test grammars' coverage of the specification
grammar (as of the version of 25 May, taken earlier this week from the
spec on SP's staging site).

I have not finished work on a parallel set of tools for negative
coverage.  (These attempt, essentially, to catalog the different ways an
input can be syntactically wrong, and check that each way is represented
in a negative test case.)

Both positive and negative coverage tools should ideally be run not only
for the spec grammar, but for at least some of the other grammars in the
test set

I have not run the choice-coverage tools on other grammars in the test
set; I will do so when and as I have a chance to do so.  I think it
would be helpful to have thorough tests, for at least a few grammars
other than the spec grammar, to make sure that processors are getting
the semantics of the spec grammar correct.

Michael

-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Friday, 27 May 2022 22:13:22 UTC