W3C home > Mailing lists > Public > public-rdf-dawg@w3.org > October to December 2006

Re: earl

From: Eric Prud'hommeaux <eric@w3.org>
Date: Mon, 2 Oct 2006 15:03:21 +0200
To: Kendall Clark <kendall@monkeyfist.com>
Cc: dawg mailing list <public-rdf-dawg@w3.org>
Message-ID: <20061002130321.GB5350@w3.org>
On Sun, Oct 01, 2006 at 09:23:23PM +0200, Eric Prud'hommeaux wrote:
> Immediately following are comments I had when reading the spec, then
> Andy's. After the quoted text are comments I had when making algae
> report EARL results.
> 
> On Mon, Sep 18, 2006 at 02:05:22AM +0200, Eric Prud'hommeaux wrote:
> > On Thu, Sep 14, 2006 at 06:33:24PM -0400, Kendall Clark wrote:
> > > 
> > > Folks,
> > > 
> > > I promised the SWCG that I'd try to find someone on DAWG to review  
> > > the EARL schema:
> > > 
> > > http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905g
> > 
> > == Example 3 ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> >   <earl:Assertion rdf:ID="#assertion">
> > should be "assertion" (kudos to emacs22 nxml-mode).
> > 
> > There are a few other mods I had to make. See the attached xslt for
> > how I extracted the examples, and the attached rdf file for what my
> > final edit was. (Ignore the html mods in the parseType="Literal" --
> > they were just so I keep the text brief.)
> > 
> > 
> > == 2.1 Assertion ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> > It may be worth associating the mode with the assertor, eg
> > 
> > <earl:Assertion rdf:ID="#assertion">
> >   <earl:assertedBy rdf:resource="#assertor" />
> >   <earl:subject rdf:resource="#subject" />
> >   <earl:test rdf:resource="#testcase" />
> >   <earl:result rdf:resource="#result" />
> >   <!-- removed mode -->
> >   <!-- earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/ -->
> > </earl:Assertion>
> > 
> > <earl:compoundAssertor rdf:ID="assertor">
> >   <dc:title xml:lang="en">Bob using Two Cool Tools</dc:title>
> >   <dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
> >   <earl:mainAssertor rdf:resource="#bob"/>
> >   <earl:helpAssertor rdf:resource="#tool"/>
> >   <earl:helpAssertor rdf:resource="#validator"/>
> >   <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/> 
> > </earl:compoundAssertor>
> > 
> > <foaf:Person rdf:ID="bob">
> >   <foaf:name>Bob B. Bobbington</foaf:name>
> >   <foaf:mbox rdf:resource="mailto:bob@example.org"/>
> >   <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
> >   <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/> 
> > </foaf:Person>
> > 
> > <earl:Software rdf:ID="tool">
> >   <dc:title xml:lang="en">Cool Tool</dc:title>
> >   <dc:description xml:lang="en">My favorite tool!</dc:description>
> >   <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> >   <dct:hasVersion>1.0.3</dct:hasVersion>
> >   <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/> 
> > </earl:Software>
> > 
> > <earl:Software rdf:ID="validator">
> >   <dc:title xml:lang="en">Cool Tool</dc:title>
> >   <dc:description xml:lang="en">My favorite tool!</dc:description>
> >   <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> >   <dct:hasVersion>1.0.3</dct:hasVersion>
> >   <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#automatic"/> 
> > </earl:Software>
> > 
> > This might be worth 10 minutes thought, but feel free to dismiss as it
> > may not have much payback.
> > 
> > 
> > == Confidence Level ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> > [[
> > Confidence in the result by an Assertor. This may be used where a tool
> > wants to assert that it has different levels of confidence about two
> > possible results (for example low confidence that a test has been
> > passed, but also high confidence that it is not applicable). The
> > values of the confidence level should be defined either as RDF values,
> > or using a datatype, in order to allow others to work interoperably
> > with them.
> > ]]
> > 
> > I don't see much value to a standardized predicate with an
> > application-specific range. I would guess you serve more needs if you
> > specify a fairly arbitrary, human-friendly range, say, 0-100, and let
> > the apps that need their own range use their own predicate in addition
> > to the standard one:
> > <earl:result rdf:about="#result">
> >   <earl:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">87</earl:confidence>
> >   <my:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">65535</earl:confidence>
> > </earl:result>
> > 
> > what is an "RDF values"? (There's only one instance of the term in the
> > spec.)
> > 
> > 
> > == 2.6.3. Instance Location ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#instancelocation?
> > [[
> > An Instance Location is a an area within the Test Subject that
> > triggered or primarily influenced the Test Result. These areas could
> > be described using a variety of different types of pointers that re
> > grouped by Pointer Collection classes. Each instance of such a
> > collection must describe exactly one occurence of an area.
> > ]]
> > 
> > Readers could use more guidance on "area". Perhaps you get the most
> > bang for the buck by saying it's logical section of a document
> > identifiable by at least one of these addressing schemes.
> > 
> > Hard case: identifying the location of a cognitive barrier introduced
> > by the intereaction of a flowed paragraph and the picture it's flowing
> > around. Worth it? I dunno. 10 minutes thought by the experts.
> > 
> > [[
> > earl:htmlPointer
> >     An HTML Pointer string expression
> > ]]
> > 
> > Is that a URL reference (#label thingy)?
> > 
> > I don't think you need both earl:charSnippet and earl:byteSnippet.
> > Rather, I think that these two identify quoted-printable and base64
> > RFC2045  6 Content-Transfer-Encodings. Please consider merging them
> > into a earl:snippit property with an object with an
> > ear:encodingMechanism with values like #quoted-printable, #eightBit,
> > #base64.
> > 
> > _:result earl:instance [
> >   earl:snippet [
> >     a earl:SnippetPointer ;
> >     earl:encodingMechanism smtp:base64 ;
> >     earl:content "A32F48ED3F2" ] ;
> >   earl:snippet [
> >     a earl:SnippetPointer ;
> >     earl:encodingMechanism smtp:quoted-printable ;
> >     earl:content "Hi mom!" ] ;
> > ] .
> > 
> > 
> > == 2.8. Web Content ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#webcontent>
> > [[
> > earl:httpResponse
> >     HTTP response (except the actual payload of the date) sent by the
> >     server to the client
> > ]]
> > 
> > s/date/data/
> > rfc2616 says that a response is made of:
> >        Response      = Status-Line               ; Section 6.1
> >                        *(( general-header        ; Section 4.5
> >                         | response-header        ; Section 6.2
> >                         | entity-header ) CRLF)  ; Section 7.1
> >                        CRLF
> >                        [ message-body ]          ; Section 7.2
> > Infortunately, there's no convenient "headers" production, but you can
> > s/payload/message-body/ in order to tie back to the RFC terminology.
> > (I assume you you want the Status-Line included.)
> > 
> > [[
> > Note: an instance of the Web Content class may include several
> > request/response sequences to document the content and language
> > negotiation that took place before the content was finally sent.
> > ]]
> > Ooo, attacking the non-REST interactions. That *is* ambitious.
> > Is that a requirement? You can never be sure you've captured enough in
> > your log for someone to duplicated the experience, even if they can go
> > back in time. If you want to give folks a rough tool that they can use
> > to captures *some* of these, I'd get rid of the dc:format (or tie it
> > redundantly to the indivitual requests and responses), and make the 
> > 
> > [ a earl:TestSubject ;
> >   earl:messageSequence ( [ Server "example.org" ; 
> > 			   port "80" ; 
> > 			   request "GET /loginPage..." ; 
> > 			   response "200..." ]
> > 			 [ server "example.org" ; 
> > 			   port "80" ; 
> > 			   request "GET /loginPage...?name=ericP..." ; 
> > 			   response "200...cookie..." ]
> > 			 [ server "example.org" ; 
> > 			   port "80" ; 
> > 			   request "GET /service...cookie..." ; 
> > 			   response "200..." ] ) ] .
> > for documents that can't be identified by URI, and 
> > [ a earl:TestSubject ;
> >   earl:uri <http://example.org/service> ]
> > for documents that can be.
> > 
> > 2.8 also could use an example.
> > 
> > 
> > == Query-ability ==
> > 
> > Looking at this from the SPARQL perspective ,this is all trivially
> > query-able. For example, from the attached data extracted from the
> > examples wiht the attached earl-ex.xslt , I queried
> > 
> > [[
> > PREFIX earl: <http://www.w3.org/WAI/ER/EARL/nmg-strawman#>
> > PREFIX dc: <http://purl.org/dc/elements/1.1/>
> > PREFIX foaf: <http://xmlns.com/foaf/0.1/>
> > SELECT ?name ?title ?val ?conf
> >  WHERE { ?assert earl:subject [ dc:title ?title ] ; 
> > 		 earl:test [ dc:identifier "http://example.org/tests/html/#282" ] ; 
> > 		 earl:result [ earl:validity ?val ; earl:confidence ?conf ] ; 
> > 		 earl:assertedBy [ earl:mainAssertor [ foaf:name ?name ] ] }
> > ]]
> > 
> > and got back:
> > 
> > +-------------------+--------------+---------+----+
> > |               name|         title|      val|conf|
> > |-------------------|--------------|---------|----|
> > |"Bob B. Bobbington"|"Login Applet"|earl:fail|  87|
> > +-------------------+--------------+---------+----+
> > 
> > 
> > == General Comments ==
> > 
> > I was impressed by this spec, though you tackle some difficult stuff
> > near the end. Have fun!
> > 
> > 
> > == Mapping to DAWG tests ==
> > should be done soon
> > -- 
> > -eric
> > 
> > home-office: +1.617.395.1213 (usually 900-2300 CET)
> > 	    +33.1.45.35.62.14
> > cell:       +33.6.73.84.87.26
> > 
> > (eric@w3.org)
> > Feel free to forward this message to any list for any purpose other than
> > email address distribution.
> 
> > <?xml version='1.0' ?>
> > <xsl:transform version="1.0" 
> >   xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
> >   xmlns:html="http://www.w3.org/1999/xhtml">
> >   <xsl:output method="text" encoding="utf-8" indent="no" />
> >   
> >   <xsl:template match="/">
> >     <xsl:text>&lt;rdf:RDF
> > 	 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
> >          xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
> >          xmlns:dc="http://purl.org/dc/elements/1.1/"
> > 	 xmlns:dct="http://purl.org/dc/terms/"
> >          xmlns:foaf="http://xmlns.com/foaf/0.1/"&gt;
> > </xsl:text>
> >       <xsl:apply-templates />
> >     <xsl:text>
> > &lt;/rdf:RDF&gt;
> > </xsl:text>
> >   </xsl:template>
> > 
> >   <xsl:template match="//html:pre[@class='schema']"/>
> > 
> >   <xsl:template match="//html:pre/html:code">
> >       <xsl:value-of select="." />
> >     <xsl:text>
> > </xsl:text>
> >   </xsl:template>
> > 
> >   <xsl:template match="text()"/>
> > 
> > </xsl:transform>
> 
> 
> 
> 
> ========================================================
> ===== Andy's comments on using EARL for DAWG tests =====
> 
> 
> On Mon, Sep 18, 2006 at 10:01:31AM +0100, Seaborne, Andy wrote:
> > 
> > 
> > 
> > Kendall Clark wrote:
> >  > Folks,
> >  >
> >  > I should have mentioned that the point of the review of EARL is
> >  > to see if it's feasible to use EARL for our test suite work.
> >  >
> >  > Kendall
> > 
> > I had a look at EARL, focusing on the design intent as a fit for DAWG.  This
> > is not a detailed technical review [*].
> > Based on the EARL editors' working draft. [EARL]
> > 
> > ==== Summary
> > 
> > EARL complements the DAWG test suite work [DAWG_Test]; it is applicable to 
> > the
> > recording of DAWG test results for the SPARQL implementation report.  It's 
> > not
> > a replacement for the current test suite materials.
> > 
> > ==== EARL
> > 
> > EARL records assertions about tests: claims about tests, who reported the 
> > test
> > results, what the outcome was reported as.  This is important for the
> > implementation reports.  An outcome is more of the style "pass/fail".
> > 
> > EARL can provide the vocabulary for test reporting; the role of EARL is to
> > record who has tested what, and with what outcome.  The DAWG test suite work
> > is describes what some SPARQL test actually is.  We could use EARL for
> > recording implementation reports against the DAWG test suite deliverable.
> > 
> > It would be more useful to use EARL to record the outcomes of each DAWG 
> > test,
> > rather than a blanket result for the whole test suite or whole DAWG manifest
> > file.  This would require some rework of the DAWG test suites.
> > 
> > ==== DAWG Test Manifest
> > 
> > The DAWG test [DT] manifest describes tests as actions and results - it
> > records what the test is and what results are expected.  A DAWG test passes 
> > if
> > exactly the same results as noted are obtained.  There is no way to record 
> > the
> > outcome of a test, only the expected results.
> > 
> > The DAWG test vocabulary is split into two - a manifest is an ordered list 
> > of
> > "tests" where a test has an action and a result (query independent).  For
> > query, an action is commonly a data file and a query, and the result is a
> > result set (various encodings) or a graph.
> > 
> > == Modelling: Similarities and differences
> > 
> > Loosely, the relationships could be:
> > 
> > EARL test subject == system being reported
> > EARL earl:TestCase == DAWG test? Or one set of tests
> > earl:TestRequirement might capture this.
> > 
> > EARL test criteria: DAWG only has "pass" - we've talked about negative tests
> > as well for syntax.
> > 
> > ==== Overlap
> > 
> > DAWG puts the test label with the test: EARL puts one in the test case and
> > points (dc:identifier) to the test.  The use of dct:isPartOf/dct:hasPart  vs
> > DAWG manifest files would need to be sorted out.
> > 
> > ==== Comments
> > 
> > EARL uses instance locations within a test subject.
> > 
> > DAWG tests should have IRIs (currently, they are commonly rooted at a blank
> > node in a list) as fragments from the manifest.  Then EARL could have a 
> > point
> > directly at it as a pointer type.
> > 
> > ==== Other
> > 
> > [*] The editor's draft contains a copy of the schema and also a very wide
> > table which do not print on portrait.
> > 
> > 	Andy
> > 
> > [EARL]
> > http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905
> > 
> > [DT]
> > http://www.w3.org/2001/sw/DataAccess/tests/README.html
> 
> 
> 
> ============================================
> ===== making algae report EARL results =====
> 
> 
> The object of the earl:result arc appears to be an earl:result object
> (I expected earl:Result). Also, I didn't see where that was specified.
> I will use "Result" in the rest of this report.
> 
> It seems there ought to be a prescribed dc:title and dc:description
> for a Result had a validity of earl:pass.
> 
> My algae code had a mode where it tests itself. Thus, the earl:subject
> of the assertion was the same as the earl:helpAssertor. I concluded
> that even when they are not the same piece of code, they ought to be
> the same class. Thus, I added foaf:homepage and dct:hasVersion to the
> earl:subject and dc:date to the earl:helpAssertor.
> 
> @@TODO@@ I didn't know how to use partOf, but I didn't do much
> research on current practice.
> 
> In general, it was pretty easy to see how to use the earl stuff. After
> considering Andy's
> > EARL earl:TestCase == DAWG test? Or one set of tests
> comment for a while and came to the conclusion that, as the result of
> a test run, there is an earl:test that has a dc:title and
> dc:description corresponding to the mf:name and rdfs:comment of a dawg
> test. Pictorially (perhaps, confusingly):
> 
> DAWG					EARL
> 
> dawg:manifest			earl:assertion
>     mf:entries			    earl:test  [
>     (					 dc:identifier  |?id|;
>      [ mf:name     |?name|;              dc:title   |?name|;
>        rdfs:comment                      dc:description
>                  |?commnt|                          |?comment|
>        ... dawg:stuff...
>     ]).                             ] .
> 
> The ?name and ?comment came from the DAWG test entry, and the ?id was
> <http://www.w3.org/2001/sw/DataAccess/tests/# + ?name> .
> This leaves the EARL data talking about the above resources, while the
> DAWG tests don't identify it. It may well be better to migrate DAWG
> tests to use the EARL structure, This would also help us correlate the
> test names with web resources like
>   <http://www.w3.org/2001/sw/DataAccess/tests/#testrdfSemantics-001>
> 
> Attached is the output of a run. I note the turtle serializer problems
> at the bottom and will look at them later. The stuff at the top may be
> enough to give you and idea what the data looked like.

Fixed the N3Serializer. See new (and pleasantly readable) N3.
-- 
-eric

home-office: +1.617.395.1213 (usually 900-2300 CET)
	    +33.1.45.35.62.14
cell:       +33.6.73.84.87.26

(eric@w3.org)
Feel free to forward this message to any list for any purpose other than
email address distribution.


rdfSemantics-earl-sample.png
(image/png attachment: rdfSemantics-earl-sample.png)

Received on Monday, 2 October 2006 13:02:28 GMT

This archive was generated by hypermail 2.3.1 : Tuesday, 26 March 2013 16:15:27 GMT