[Testing] Alternative approach to the test manifest structure

All,

After playing around with the schemas over the weekend, with a view to
integrating them into my server implementation to validate the incoming
annotations, I ran into some issues:

* The tests are designed for humans to read the error message, not for
machines to process the results ... some tests are okay to fail validation,
some aren't.

* There doesn't seem to be a way to descend into the referenced resources
automatically.  You need to run the specific resource tests against the
specific resource by hand.

* The processing patterns of the single with break or skip seems like it
could be extended ...


So what do people think about the following, if it's not too late to change
things:

* Continue with atomic tests for presence of a property, and then a
separate one for the value of it
* But do that in the framework testing "manifest" by testing for
failure/success of the validation.

For example, an automated system could descend into a SpecificResource as
the body by:

* Test that body exists
  -- OnFail:  Warn (body is a SHOULD)
  -- OnSuccess:
      * Determine type of body
          -- uri?  OnSuccess:  Test URI-ness & goto next set of tests
                      OnFail: SpecificResource?
                              OnSuccess:  Descend into SpecificResource
tests
                              OnFail: TextualBody?

And so forth.
The success/fail would be by the same $ref approach as the schema includes,
and offset from the schema itself, so it can be reused in different parts
of the overall set of tests.

This would let us compose features at whatever level we think is
appropriate for reporting, and give a good validation suite for the model
that can be used completely programmatically by implementing the
success/fail runner.
[I did this runner already in python, it's a pretty easy piece of code, as
one might expect]

Thoughts?

Rob

-- 
Rob Sanderson
Semantic Architect
The Getty Trust
Los Angeles, CA 90049