Re: more on earl

Kendall Clark wrote:
  > Folks,
  >
  > I should have mentioned that the point of the review of EARL is
  > to see if it's feasible to use EARL for our test suite work.
  >
  > Kendall

I had a look at EARL, focusing on the design intent as a fit for DAWG.  This
is not a detailed technical review [*].
Based on the EARL editors' working draft. [EARL]

==== Summary

EARL complements the DAWG test suite work [DAWG_Test]; it is applicable to the
recording of DAWG test results for the SPARQL implementation report.  It's not
a replacement for the current test suite materials.

==== EARL

EARL records assertions about tests: claims about tests, who reported the test
results, what the outcome was reported as.  This is important for the
implementation reports.  An outcome is more of the style "pass/fail".

EARL can provide the vocabulary for test reporting; the role of EARL is to
record who has tested what, and with what outcome.  The DAWG test suite work
is describes what some SPARQL test actually is.  We could use EARL for
recording implementation reports against the DAWG test suite deliverable.

It would be more useful to use EARL to record the outcomes of each DAWG test,
rather than a blanket result for the whole test suite or whole DAWG manifest
file.  This would require some rework of the DAWG test suites.

==== DAWG Test Manifest

The DAWG test [DT] manifest describes tests as actions and results - it
records what the test is and what results are expected.  A DAWG test passes if
exactly the same results as noted are obtained.  There is no way to record the
outcome of a test, only the expected results.

The DAWG test vocabulary is split into two - a manifest is an ordered list of
"tests" where a test has an action and a result (query independent).  For
query, an action is commonly a data file and a query, and the result is a
result set (various encodings) or a graph.

== Modelling: Similarities and differences

Loosely, the relationships could be:

EARL test subject == system being reported
EARL earl:TestCase == DAWG test? Or one set of tests
earl:TestRequirement might capture this.

EARL test criteria: DAWG only has "pass" - we've talked about negative tests
as well for syntax.

==== Overlap

DAWG puts the test label with the test: EARL puts one in the test case and
points (dc:identifier) to the test.  The use of dct:isPartOf/dct:hasPart  vs
DAWG manifest files would need to be sorted out.

==== Comments

EARL uses instance locations within a test subject.

DAWG tests should have IRIs (currently, they are commonly rooted at a blank
node in a list) as fragments from the manifest.  Then EARL could have a point
directly at it as a pointer type.

==== Other

[*] The editor's draft contains a copy of the schema and also a very wide
table which do not print on portrait.

	Andy

[EARL]
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905

[DT]
http://www.w3.org/2001/sw/DataAccess/tests/README.html

Received on Monday, 18 September 2006 09:02:00 UTC