- From: Eric Prud'hommeaux <eric@w3.org>
- Date: Sun, 1 Oct 2006 21:23:23 +0200
- To: Kendall Clark <kendall@monkeyfist.com>
- Cc: dawg mailing list <public-rdf-dawg@w3.org>
- Message-ID: <20061001192323.GA5351@w3.org>
Immediately following are comments I had when reading the spec, then
Andy's. After the quoted text are comments I had when making algae
report EARL results.
On Mon, Sep 18, 2006 at 02:05:22AM +0200, Eric Prud'hommeaux wrote:
> On Thu, Sep 14, 2006 at 06:33:24PM -0400, Kendall Clark wrote:
> >
> > Folks,
> >
> > I promised the SWCG that I'd try to find someone on DAWG to review
> > the EARL schema:
> >
> > http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905g
>
> == Example 3 ==
> <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> <earl:Assertion rdf:ID="#assertion">
> should be "assertion" (kudos to emacs22 nxml-mode).
>
> There are a few other mods I had to make. See the attached xslt for
> how I extracted the examples, and the attached rdf file for what my
> final edit was. (Ignore the html mods in the parseType="Literal" --
> they were just so I keep the text brief.)
>
>
> == 2.1 Assertion ==
> <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> It may be worth associating the mode with the assertor, eg
>
> <earl:Assertion rdf:ID="#assertion">
> <earl:assertedBy rdf:resource="#assertor" />
> <earl:subject rdf:resource="#subject" />
> <earl:test rdf:resource="#testcase" />
> <earl:result rdf:resource="#result" />
> <!-- removed mode -->
> <!-- earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/ -->
> </earl:Assertion>
>
> <earl:compoundAssertor rdf:ID="assertor">
> <dc:title xml:lang="en">Bob using Two Cool Tools</dc:title>
> <dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
> <earl:mainAssertor rdf:resource="#bob"/>
> <earl:helpAssertor rdf:resource="#tool"/>
> <earl:helpAssertor rdf:resource="#validator"/>
> <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/>
> </earl:compoundAssertor>
>
> <foaf:Person rdf:ID="bob">
> <foaf:name>Bob B. Bobbington</foaf:name>
> <foaf:mbox rdf:resource="mailto:bob@example.org"/>
> <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
> <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/>
> </foaf:Person>
>
> <earl:Software rdf:ID="tool">
> <dc:title xml:lang="en">Cool Tool</dc:title>
> <dc:description xml:lang="en">My favorite tool!</dc:description>
> <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> <dct:hasVersion>1.0.3</dct:hasVersion>
> <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/>
> </earl:Software>
>
> <earl:Software rdf:ID="validator">
> <dc:title xml:lang="en">Cool Tool</dc:title>
> <dc:description xml:lang="en">My favorite tool!</dc:description>
> <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> <dct:hasVersion>1.0.3</dct:hasVersion>
> <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#automatic"/>
> </earl:Software>
>
> This might be worth 10 minutes thought, but feel free to dismiss as it
> may not have much payback.
>
>
> == Confidence Level ==
> <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> [[
> Confidence in the result by an Assertor. This may be used where a tool
> wants to assert that it has different levels of confidence about two
> possible results (for example low confidence that a test has been
> passed, but also high confidence that it is not applicable). The
> values of the confidence level should be defined either as RDF values,
> or using a datatype, in order to allow others to work interoperably
> with them.
> ]]
>
> I don't see much value to a standardized predicate with an
> application-specific range. I would guess you serve more needs if you
> specify a fairly arbitrary, human-friendly range, say, 0-100, and let
> the apps that need their own range use their own predicate in addition
> to the standard one:
> <earl:result rdf:about="#result">
> <earl:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">87</earl:confidence>
> <my:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">65535</earl:confidence>
> </earl:result>
>
> what is an "RDF values"? (There's only one instance of the term in the
> spec.)
>
>
> == 2.6.3. Instance Location ==
> <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#instancelocation?
> [[
> An Instance Location is a an area within the Test Subject that
> triggered or primarily influenced the Test Result. These areas could
> be described using a variety of different types of pointers that re
> grouped by Pointer Collection classes. Each instance of such a
> collection must describe exactly one occurence of an area.
> ]]
>
> Readers could use more guidance on "area". Perhaps you get the most
> bang for the buck by saying it's logical section of a document
> identifiable by at least one of these addressing schemes.
>
> Hard case: identifying the location of a cognitive barrier introduced
> by the intereaction of a flowed paragraph and the picture it's flowing
> around. Worth it? I dunno. 10 minutes thought by the experts.
>
> [[
> earl:htmlPointer
> An HTML Pointer string expression
> ]]
>
> Is that a URL reference (#label thingy)?
>
> I don't think you need both earl:charSnippet and earl:byteSnippet.
> Rather, I think that these two identify quoted-printable and base64
> RFC2045 § 6 Content-Transfer-Encodings. Please consider merging them
> into a earl:snippit property with an object with an
> ear:encodingMechanism with values like #quoted-printable, #eightBit,
> #base64.
>
> _:result earl:instance [
> earl:snippet [
> a earl:SnippetPointer ;
> earl:encodingMechanism smtp:base64 ;
> earl:content "A32F48ED3F2" ] ;
> earl:snippet [
> a earl:SnippetPointer ;
> earl:encodingMechanism smtp:quoted-printable ;
> earl:content "Hi mom!" ] ;
> ] .
>
>
> == 2.8. Web Content ==
> <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#webcontent>
> [[
> earl:httpResponse
> HTTP response (except the actual payload of the date) sent by the
> server to the client
> ]]
>
> s/date/data/
> rfc2616 says that a response is made of:
> Response = Status-Line ; Section 6.1
> *(( general-header ; Section 4.5
> | response-header ; Section 6.2
> | entity-header ) CRLF) ; Section 7.1
> CRLF
> [ message-body ] ; Section 7.2
> Infortunately, there's no convenient "headers" production, but you can
> s/payload/message-body/ in order to tie back to the RFC terminology.
> (I assume you you want the Status-Line included.)
>
> [[
> Note: an instance of the Web Content class may include several
> request/response sequences to document the content and language
> negotiation that took place before the content was finally sent.
> ]]
> Ooo, attacking the non-REST interactions. That *is* ambitious.
> Is that a requirement? You can never be sure you've captured enough in
> your log for someone to duplicated the experience, even if they can go
> back in time. If you want to give folks a rough tool that they can use
> to captures *some* of these, I'd get rid of the dc:format (or tie it
> redundantly to the indivitual requests and responses), and make the
>
> [ a earl:TestSubject ;
> earl:messageSequence ( [ Server "example.org" ;
> port "80" ;
> request "GET /loginPage..." ;
> response "200..." ]
> [ server "example.org" ;
> port "80" ;
> request "GET /loginPage...?name=ericP..." ;
> response "200...cookie..." ]
> [ server "example.org" ;
> port "80" ;
> request "GET /service...cookie..." ;
> response "200..." ] ) ] .
> for documents that can't be identified by URI, and
> [ a earl:TestSubject ;
> earl:uri <http://example.org/service> ]
> for documents that can be.
>
> 2.8 also could use an example.
>
>
> == Query-ability ==
>
> Looking at this from the SPARQL perspective ,this is all trivially
> query-able. For example, from the attached data extracted from the
> examples wiht the attached earl-ex.xslt , I queried
>
> [[
> PREFIX earl: <http://www.w3.org/WAI/ER/EARL/nmg-strawman#>
> PREFIX dc: <http://purl.org/dc/elements/1.1/>
> PREFIX foaf: <http://xmlns.com/foaf/0.1/>
> SELECT ?name ?title ?val ?conf
> WHERE { ?assert earl:subject [ dc:title ?title ] ;
> earl:test [ dc:identifier "http://example.org/tests/html/#282" ] ;
> earl:result [ earl:validity ?val ; earl:confidence ?conf ] ;
> earl:assertedBy [ earl:mainAssertor [ foaf:name ?name ] ] }
> ]]
>
> and got back:
>
> +-------------------+--------------+---------+----+
> | name| title| val|conf|
> |-------------------|--------------|---------|----|
> |"Bob B. Bobbington"|"Login Applet"|earl:fail| 87|
> +-------------------+--------------+---------+----+
>
>
> == General Comments ==
>
> I was impressed by this spec, though you tackle some difficult stuff
> near the end. Have fun!
>
>
> == Mapping to DAWG tests ==
> should be done soon
> --
> -eric
>
> home-office: +1.617.395.1213 (usually 900-2300 CET)
> +33.1.45.35.62.14
> cell: +33.6.73.84.87.26
>
> (eric@w3.org)
> Feel free to forward this message to any list for any purpose other than
> email address distribution.
> <?xml version='1.0' ?>
> <xsl:transform version="1.0"
> xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
> xmlns:html="http://www.w3.org/1999/xhtml">
> <xsl:output method="text" encoding="utf-8" indent="no" />
>
> <xsl:template match="/">
> <xsl:text><rdf:RDF
> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
> xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
> xmlns:dc="http://purl.org/dc/elements/1.1/"
> xmlns:dct="http://purl.org/dc/terms/"
> xmlns:foaf="http://xmlns.com/foaf/0.1/">
> </xsl:text>
> <xsl:apply-templates />
> <xsl:text>
> </rdf:RDF>
> </xsl:text>
> </xsl:template>
>
> <xsl:template match="//html:pre[@class='schema']"/>
>
> <xsl:template match="//html:pre/html:code">
> <xsl:value-of select="." />
> <xsl:text>
> </xsl:text>
> </xsl:template>
>
> <xsl:template match="text()"/>
>
> </xsl:transform>
========================================================
===== Andy's comments on using EARL for DAWG tests =====
On Mon, Sep 18, 2006 at 10:01:31AM +0100, Seaborne, Andy wrote:
>
>
>
> Kendall Clark wrote:
> > Folks,
> >
> > I should have mentioned that the point of the review of EARL is
> > to see if it's feasible to use EARL for our test suite work.
> >
> > Kendall
>
> I had a look at EARL, focusing on the design intent as a fit for DAWG. This
> is not a detailed technical review [*].
> Based on the EARL editors' working draft. [EARL]
>
> ==== Summary
>
> EARL complements the DAWG test suite work [DAWG_Test]; it is applicable to
> the
> recording of DAWG test results for the SPARQL implementation report. It's
> not
> a replacement for the current test suite materials.
>
> ==== EARL
>
> EARL records assertions about tests: claims about tests, who reported the
> test
> results, what the outcome was reported as. This is important for the
> implementation reports. An outcome is more of the style "pass/fail".
>
> EARL can provide the vocabulary for test reporting; the role of EARL is to
> record who has tested what, and with what outcome. The DAWG test suite work
> is describes what some SPARQL test actually is. We could use EARL for
> recording implementation reports against the DAWG test suite deliverable.
>
> It would be more useful to use EARL to record the outcomes of each DAWG
> test,
> rather than a blanket result for the whole test suite or whole DAWG manifest
> file. This would require some rework of the DAWG test suites.
>
> ==== DAWG Test Manifest
>
> The DAWG test [DT] manifest describes tests as actions and results - it
> records what the test is and what results are expected. A DAWG test passes
> if
> exactly the same results as noted are obtained. There is no way to record
> the
> outcome of a test, only the expected results.
>
> The DAWG test vocabulary is split into two - a manifest is an ordered list
> of
> "tests" where a test has an action and a result (query independent). For
> query, an action is commonly a data file and a query, and the result is a
> result set (various encodings) or a graph.
>
> == Modelling: Similarities and differences
>
> Loosely, the relationships could be:
>
> EARL test subject == system being reported
> EARL earl:TestCase == DAWG test? Or one set of tests
> earl:TestRequirement might capture this.
>
> EARL test criteria: DAWG only has "pass" - we've talked about negative tests
> as well for syntax.
>
> ==== Overlap
>
> DAWG puts the test label with the test: EARL puts one in the test case and
> points (dc:identifier) to the test. The use of dct:isPartOf/dct:hasPart vs
> DAWG manifest files would need to be sorted out.
>
> ==== Comments
>
> EARL uses instance locations within a test subject.
>
> DAWG tests should have IRIs (currently, they are commonly rooted at a blank
> node in a list) as fragments from the manifest. Then EARL could have a
> point
> directly at it as a pointer type.
>
> ==== Other
>
> [*] The editor's draft contains a copy of the schema and also a very wide
> table which do not print on portrait.
>
> Andy
>
> [EARL]
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905
>
> [DT]
> http://www.w3.org/2001/sw/DataAccess/tests/README.html
============================================
===== making algae report EARL results =====
The object of the earl:result arc appears to be an earl:result object
(I expected earl:Result). Also, I didn't see where that was specified.
I will use "Result" in the rest of this report.
It seems there ought to be a prescribed dc:title and dc:description
for a Result had a validity of earl:pass.
My algae code had a mode where it tests itself. Thus, the earl:subject
of the assertion was the same as the earl:helpAssertor. I concluded
that even when they are not the same piece of code, they ought to be
the same class. Thus, I added foaf:homepage and dct:hasVersion to the
earl:subject and dc:date to the earl:helpAssertor.
@@TODO@@ I didn't know how to use partOf, but I didn't do much
research on current practice.
In general, it was pretty easy to see how to use the earl stuff. After
considering Andy's
> EARL earl:TestCase == DAWG test? Or one set of tests
comment for a while and came to the conclusion that, as the result of
a test run, there is an earl:test that has a dc:title and
dc:description corresponding to the mf:name and rdfs:comment of a dawg
test. Pictorially (perhaps, confusingly):
DAWG EARL
dawg:manifest earl:assertion
mf:entries earl:test [
( dc:identifier |?id|;
[ mf:name |?name|; dc:title |?name|;
rdfs:comment dc:description
|?commnt| |?comment|
... dawg:stuff...
]). ] .
The ?name and ?comment came from the DAWG test entry, and the ?id was
<http://www.w3.org/2001/sw/DataAccess/tests/# + ?name> .
This leaves the EARL data talking about the above resources, while the
DAWG tests don't identify it. It may well be better to migrate DAWG
tests to use the EARL structure, This would also help us correlate the
test names with web resources like
<http://www.w3.org/2001/sw/DataAccess/tests/#testrdfSemantics-001>
Attached is the output of a run. I note the turtle serializer problems
at the bottom and will look at them later. The stuff at the top may be
enough to give you and idea what the data looked like.
--
-eric
home-office: +1.617.395.1213 (usually 900-2300 CET)
+33.1.45.35.62.14
cell: +33.6.73.84.87.26
(eric@w3.org)
Feel free to forward this message to any list for any purpose other than
email address distribution.
Attachments
- text/plain attachment: rdfSemantics-earl.ttl
Received on Sunday, 1 October 2006 19:22:22 UTC