Bug 017: Unclear Meaning of TestRequirement

This is feedback on a Last Call Working Draft:

Evaluation and Report Language (EARL) 1.0 Schema
W3C Working Draft 10 May 2011
http://www.w3.org/TR/2011/WD-EARL10-Schema-20110510/

Specifically of § 2.4. TestCriterion Class:

http://www.w3.org/TR/2011/WD-EARL10-Schema-20110510/#TestCriterion

Where the following sentence:

"Test Requirement - a higher-level requirement that is tested by
executing one or more sub-tests."

Is unclear. What does it mean to be a “higher-level” requirement? This
metaphor could mean that the requirement has a greater importance than
other tests. It probably means, in context, that it is composed rather
than atomic, but this is not clear.

This lack of clarity seems to pervade the entire distinction between
earl:TestRequirement and earl:TestCase. For example, consider the fact
that the earl:OutcomeValue instances are obviously oriented towards
providing for earl:TestCase assertions and not earl:TestRequirement
assertions. Say that you have a TestRequirement and five of the sub
tests pass and one fails. Is that really a failure? Using earl:failed
here would be technically accurate, but not very actually informative
to the tester.

It may be that "Fail - the class of outcomes to denote failing a test.
Subclasses may include ordinal, nominal, or continuous values or
expressions." in § 2.7 allows for something like:

:fivePassesOneFail a earl:Fail.

But this is peculiar. You'd expect at least to allow properties:

:CompoundFailWithSuccesses a earl:Fail.
:passes rdfs:domain :CompoundFailWithSuccesses.
:fails rdfs:domain :CompoundFailWithSuccesses.
[ a earl:Fail; :passes 5; :fails 1 ].

(Using RDF Schema rather than OWL restrictions for demonstrative purposes only.)

Which goes even further into the realm of extensions, on which compare
Bug 006. So this once again has the problem of whether one should do [
a earl:Fail ], or [ a :CompoundFailWithSuccesses ], or [ a earl:Fail,
:CompoundFailWithSuccesses ], and even whether the new properties here
are appropriate and would be conformant. My reading is that they would
be, though they would also need a dct:title and dct:description, which
honestly seems redundant. That was broached in Bug 004.

-- 
Sean B. Palmer, http://inamidst.com/sbp/

Received on Wednesday, 11 May 2011 12:53:36 UTC