Granuarlity of Test Framework JSON API


Back in the F2F, I took the ACTION-58 [1] to find a way to get the JSON
API to expose list of test cases matching a given spec or a given
feature, rather than the semi-random set of test cases currently bundled
as "test suite".

A "feature" is a fuzzy concept, but I think it's fair to say that we
could easily bind a feature to a list of spec or a list of spec

After discussions with Robin, it appears that the API already lets one
get a list of tests cases per spec section in a given test suite.

For instance,
lists all the test cases associated with all the known section of the
DOM Element traversal spec in the DOM Element traversal test suite.

It isn't exactly what we would want:
* we would want to get all the test cases linked to a given spec or spec
sections across test suites; I filed a bug for that:

* ideally, we would also want to store the bundle of test cases for a
given feature with some additional parameters (e.g. automatically
runnable) in the framework itself; I filed another bug for that:

The second bug is more involved, so unlikely to be tackled in the short

Robin seemed more optimistic that the first one could be tackled more
rapidly, although it's not entirely clear whether he would be himself be
in a position to tackle it — I think it would be great if someone in the
group could maybe take a stab at it?

Once the first bug is fixed, we can define the bundles out-of-band, and
let our own system do the manual work of getting the relevant list of
test cases from a list of spec sections we would maintain.

I think this is a reasonable realization of my action item, so I intend
to close it unless anybody objects.



Received on Wednesday, 17 October 2012 12:32:57 UTC