- From: Eric Prud'hommeaux <eric@w3.org>
- Date: Mon, 2 Oct 2006 15:03:21 +0200
- To: Kendall Clark <kendall@monkeyfist.com>
- Cc: dawg mailing list <public-rdf-dawg@w3.org>
- Message-ID: <20061002130321.GB5350@w3.org>
On Sun, Oct 01, 2006 at 09:23:23PM +0200, Eric Prud'hommeaux wrote:
> Immediately following are comments I had when reading the spec, then
> Andy's. After the quoted text are comments I had when making algae
> report EARL results.
>
> On Mon, Sep 18, 2006 at 02:05:22AM +0200, Eric Prud'hommeaux wrote:
> > On Thu, Sep 14, 2006 at 06:33:24PM -0400, Kendall Clark wrote:
> > >
> > > Folks,
> > >
> > > I promised the SWCG that I'd try to find someone on DAWG to review
> > > the EARL schema:
> > >
> > > http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905g
> >
> > == Example 3 ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> > <earl:Assertion rdf:ID="#assertion">
> > should be "assertion" (kudos to emacs22 nxml-mode).
> >
> > There are a few other mods I had to make. See the attached xslt for
> > how I extracted the examples, and the attached rdf file for what my
> > final edit was. (Ignore the html mods in the parseType="Literal" --
> > they were just so I keep the text brief.)
> >
> >
> > == 2.1 Assertion ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> > It may be worth associating the mode with the assertor, eg
> >
> > <earl:Assertion rdf:ID="#assertion">
> > <earl:assertedBy rdf:resource="#assertor" />
> > <earl:subject rdf:resource="#subject" />
> > <earl:test rdf:resource="#testcase" />
> > <earl:result rdf:resource="#result" />
> > <!-- removed mode -->
> > <!-- earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/ -->
> > </earl:Assertion>
> >
> > <earl:compoundAssertor rdf:ID="assertor">
> > <dc:title xml:lang="en">Bob using Two Cool Tools</dc:title>
> > <dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
> > <earl:mainAssertor rdf:resource="#bob"/>
> > <earl:helpAssertor rdf:resource="#tool"/>
> > <earl:helpAssertor rdf:resource="#validator"/>
> > <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#mixed"/>
> > </earl:compoundAssertor>
> >
> > <foaf:Person rdf:ID="bob">
> > <foaf:name>Bob B. Bobbington</foaf:name>
> > <foaf:mbox rdf:resource="mailto:bob@example.org"/>
> > <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
> > <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/>
> > </foaf:Person>
> >
> > <earl:Software rdf:ID="tool">
> > <dc:title xml:lang="en">Cool Tool</dc:title>
> > <dc:description xml:lang="en">My favorite tool!</dc:description>
> > <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> > <dct:hasVersion>1.0.3</dct:hasVersion>
> > <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#semiauto"/>
> > </earl:Software>
> >
> > <earl:Software rdf:ID="validator">
> > <dc:title xml:lang="en">Cool Tool</dc:title>
> > <dc:description xml:lang="en">My favorite tool!</dc:description>
> > <foaf:homepage>http://example.org/tools/#cool</foaf:homepage>
> > <dct:hasVersion>1.0.3</dct:hasVersion>
> > <earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#automatic"/>
> > </earl:Software>
> >
> > This might be worth 10 minutes thought, but feel free to dismiss as it
> > may not have much payback.
> >
> >
> > == Confidence Level ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#example-3>
> > [[
> > Confidence in the result by an Assertor. This may be used where a tool
> > wants to assert that it has different levels of confidence about two
> > possible results (for example low confidence that a test has been
> > passed, but also high confidence that it is not applicable). The
> > values of the confidence level should be defined either as RDF values,
> > or using a datatype, in order to allow others to work interoperably
> > with them.
> > ]]
> >
> > I don't see much value to a standardized predicate with an
> > application-specific range. I would guess you serve more needs if you
> > specify a fairly arbitrary, human-friendly range, say, 0-100, and let
> > the apps that need their own range use their own predicate in addition
> > to the standard one:
> > <earl:result rdf:about="#result">
> > <earl:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">87</earl:confidence>
> > <my:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#int">65535</earl:confidence>
> > </earl:result>
> >
> > what is an "RDF values"? (There's only one instance of the term in the
> > spec.)
> >
> >
> > == 2.6.3. Instance Location ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#instancelocation?
> > [[
> > An Instance Location is a an area within the Test Subject that
> > triggered or primarily influenced the Test Result. These areas could
> > be described using a variety of different types of pointers that re
> > grouped by Pointer Collection classes. Each instance of such a
> > collection must describe exactly one occurence of an area.
> > ]]
> >
> > Readers could use more guidance on "area". Perhaps you get the most
> > bang for the buck by saying it's logical section of a document
> > identifiable by at least one of these addressing schemes.
> >
> > Hard case: identifying the location of a cognitive barrier introduced
> > by the intereaction of a flowed paragraph and the picture it's flowing
> > around. Worth it? I dunno. 10 minutes thought by the experts.
> >
> > [[
> > earl:htmlPointer
> > An HTML Pointer string expression
> > ]]
> >
> > Is that a URL reference (#label thingy)?
> >
> > I don't think you need both earl:charSnippet and earl:byteSnippet.
> > Rather, I think that these two identify quoted-printable and base64
> > RFC2045 § 6 Content-Transfer-Encodings. Please consider merging them
> > into a earl:snippit property with an object with an
> > ear:encodingMechanism with values like #quoted-printable, #eightBit,
> > #base64.
> >
> > _:result earl:instance [
> > earl:snippet [
> > a earl:SnippetPointer ;
> > earl:encodingMechanism smtp:base64 ;
> > earl:content "A32F48ED3F2" ] ;
> > earl:snippet [
> > a earl:SnippetPointer ;
> > earl:encodingMechanism smtp:quoted-printable ;
> > earl:content "Hi mom!" ] ;
> > ] .
> >
> >
> > == 2.8. Web Content ==
> > <http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905#webcontent>
> > [[
> > earl:httpResponse
> > HTTP response (except the actual payload of the date) sent by the
> > server to the client
> > ]]
> >
> > s/date/data/
> > rfc2616 says that a response is made of:
> > Response = Status-Line ; Section 6.1
> > *(( general-header ; Section 4.5
> > | response-header ; Section 6.2
> > | entity-header ) CRLF) ; Section 7.1
> > CRLF
> > [ message-body ] ; Section 7.2
> > Infortunately, there's no convenient "headers" production, but you can
> > s/payload/message-body/ in order to tie back to the RFC terminology.
> > (I assume you you want the Status-Line included.)
> >
> > [[
> > Note: an instance of the Web Content class may include several
> > request/response sequences to document the content and language
> > negotiation that took place before the content was finally sent.
> > ]]
> > Ooo, attacking the non-REST interactions. That *is* ambitious.
> > Is that a requirement? You can never be sure you've captured enough in
> > your log for someone to duplicated the experience, even if they can go
> > back in time. If you want to give folks a rough tool that they can use
> > to captures *some* of these, I'd get rid of the dc:format (or tie it
> > redundantly to the indivitual requests and responses), and make the
> >
> > [ a earl:TestSubject ;
> > earl:messageSequence ( [ Server "example.org" ;
> > port "80" ;
> > request "GET /loginPage..." ;
> > response "200..." ]
> > [ server "example.org" ;
> > port "80" ;
> > request "GET /loginPage...?name=ericP..." ;
> > response "200...cookie..." ]
> > [ server "example.org" ;
> > port "80" ;
> > request "GET /service...cookie..." ;
> > response "200..." ] ) ] .
> > for documents that can't be identified by URI, and
> > [ a earl:TestSubject ;
> > earl:uri <http://example.org/service> ]
> > for documents that can be.
> >
> > 2.8 also could use an example.
> >
> >
> > == Query-ability ==
> >
> > Looking at this from the SPARQL perspective ,this is all trivially
> > query-able. For example, from the attached data extracted from the
> > examples wiht the attached earl-ex.xslt , I queried
> >
> > [[
> > PREFIX earl: <http://www.w3.org/WAI/ER/EARL/nmg-strawman#>
> > PREFIX dc: <http://purl.org/dc/elements/1.1/>
> > PREFIX foaf: <http://xmlns.com/foaf/0.1/>
> > SELECT ?name ?title ?val ?conf
> > WHERE { ?assert earl:subject [ dc:title ?title ] ;
> > earl:test [ dc:identifier "http://example.org/tests/html/#282" ] ;
> > earl:result [ earl:validity ?val ; earl:confidence ?conf ] ;
> > earl:assertedBy [ earl:mainAssertor [ foaf:name ?name ] ] }
> > ]]
> >
> > and got back:
> >
> > +-------------------+--------------+---------+----+
> > | name| title| val|conf|
> > |-------------------|--------------|---------|----|
> > |"Bob B. Bobbington"|"Login Applet"|earl:fail| 87|
> > +-------------------+--------------+---------+----+
> >
> >
> > == General Comments ==
> >
> > I was impressed by this spec, though you tackle some difficult stuff
> > near the end. Have fun!
> >
> >
> > == Mapping to DAWG tests ==
> > should be done soon
> > --
> > -eric
> >
> > home-office: +1.617.395.1213 (usually 900-2300 CET)
> > +33.1.45.35.62.14
> > cell: +33.6.73.84.87.26
> >
> > (eric@w3.org)
> > Feel free to forward this message to any list for any purpose other than
> > email address distribution.
>
> > <?xml version='1.0' ?>
> > <xsl:transform version="1.0"
> > xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
> > xmlns:html="http://www.w3.org/1999/xhtml">
> > <xsl:output method="text" encoding="utf-8" indent="no" />
> >
> > <xsl:template match="/">
> > <xsl:text><rdf:RDF
> > xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
> > xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
> > xmlns:dc="http://purl.org/dc/elements/1.1/"
> > xmlns:dct="http://purl.org/dc/terms/"
> > xmlns:foaf="http://xmlns.com/foaf/0.1/">
> > </xsl:text>
> > <xsl:apply-templates />
> > <xsl:text>
> > </rdf:RDF>
> > </xsl:text>
> > </xsl:template>
> >
> > <xsl:template match="//html:pre[@class='schema']"/>
> >
> > <xsl:template match="//html:pre/html:code">
> > <xsl:value-of select="." />
> > <xsl:text>
> > </xsl:text>
> > </xsl:template>
> >
> > <xsl:template match="text()"/>
> >
> > </xsl:transform>
>
>
>
>
> ========================================================
> ===== Andy's comments on using EARL for DAWG tests =====
>
>
> On Mon, Sep 18, 2006 at 10:01:31AM +0100, Seaborne, Andy wrote:
> >
> >
> >
> > Kendall Clark wrote:
> > > Folks,
> > >
> > > I should have mentioned that the point of the review of EARL is
> > > to see if it's feasible to use EARL for our test suite work.
> > >
> > > Kendall
> >
> > I had a look at EARL, focusing on the design intent as a fit for DAWG. This
> > is not a detailed technical review [*].
> > Based on the EARL editors' working draft. [EARL]
> >
> > ==== Summary
> >
> > EARL complements the DAWG test suite work [DAWG_Test]; it is applicable to
> > the
> > recording of DAWG test results for the SPARQL implementation report. It's
> > not
> > a replacement for the current test suite materials.
> >
> > ==== EARL
> >
> > EARL records assertions about tests: claims about tests, who reported the
> > test
> > results, what the outcome was reported as. This is important for the
> > implementation reports. An outcome is more of the style "pass/fail".
> >
> > EARL can provide the vocabulary for test reporting; the role of EARL is to
> > record who has tested what, and with what outcome. The DAWG test suite work
> > is describes what some SPARQL test actually is. We could use EARL for
> > recording implementation reports against the DAWG test suite deliverable.
> >
> > It would be more useful to use EARL to record the outcomes of each DAWG
> > test,
> > rather than a blanket result for the whole test suite or whole DAWG manifest
> > file. This would require some rework of the DAWG test suites.
> >
> > ==== DAWG Test Manifest
> >
> > The DAWG test [DT] manifest describes tests as actions and results - it
> > records what the test is and what results are expected. A DAWG test passes
> > if
> > exactly the same results as noted are obtained. There is no way to record
> > the
> > outcome of a test, only the expected results.
> >
> > The DAWG test vocabulary is split into two - a manifest is an ordered list
> > of
> > "tests" where a test has an action and a result (query independent). For
> > query, an action is commonly a data file and a query, and the result is a
> > result set (various encodings) or a graph.
> >
> > == Modelling: Similarities and differences
> >
> > Loosely, the relationships could be:
> >
> > EARL test subject == system being reported
> > EARL earl:TestCase == DAWG test? Or one set of tests
> > earl:TestRequirement might capture this.
> >
> > EARL test criteria: DAWG only has "pass" - we've talked about negative tests
> > as well for syntax.
> >
> > ==== Overlap
> >
> > DAWG puts the test label with the test: EARL puts one in the test case and
> > points (dc:identifier) to the test. The use of dct:isPartOf/dct:hasPart vs
> > DAWG manifest files would need to be sorted out.
> >
> > ==== Comments
> >
> > EARL uses instance locations within a test subject.
> >
> > DAWG tests should have IRIs (currently, they are commonly rooted at a blank
> > node in a list) as fragments from the manifest. Then EARL could have a
> > point
> > directly at it as a pointer type.
> >
> > ==== Other
> >
> > [*] The editor's draft contains a copy of the schema and also a very wide
> > table which do not print on portrait.
> >
> > Andy
> >
> > [EARL]
> > http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20060905
> >
> > [DT]
> > http://www.w3.org/2001/sw/DataAccess/tests/README.html
>
>
>
> ============================================
> ===== making algae report EARL results =====
>
>
> The object of the earl:result arc appears to be an earl:result object
> (I expected earl:Result). Also, I didn't see where that was specified.
> I will use "Result" in the rest of this report.
>
> It seems there ought to be a prescribed dc:title and dc:description
> for a Result had a validity of earl:pass.
>
> My algae code had a mode where it tests itself. Thus, the earl:subject
> of the assertion was the same as the earl:helpAssertor. I concluded
> that even when they are not the same piece of code, they ought to be
> the same class. Thus, I added foaf:homepage and dct:hasVersion to the
> earl:subject and dc:date to the earl:helpAssertor.
>
> @@TODO@@ I didn't know how to use partOf, but I didn't do much
> research on current practice.
>
> In general, it was pretty easy to see how to use the earl stuff. After
> considering Andy's
> > EARL earl:TestCase == DAWG test? Or one set of tests
> comment for a while and came to the conclusion that, as the result of
> a test run, there is an earl:test that has a dc:title and
> dc:description corresponding to the mf:name and rdfs:comment of a dawg
> test. Pictorially (perhaps, confusingly):
>
> DAWG EARL
>
> dawg:manifest earl:assertion
> mf:entries earl:test [
> ( dc:identifier |?id|;
> [ mf:name |?name|; dc:title |?name|;
> rdfs:comment dc:description
> |?commnt| |?comment|
> ... dawg:stuff...
> ]). ] .
>
> The ?name and ?comment came from the DAWG test entry, and the ?id was
> <http://www.w3.org/2001/sw/DataAccess/tests/# + ?name> .
> This leaves the EARL data talking about the above resources, while the
> DAWG tests don't identify it. It may well be better to migrate DAWG
> tests to use the EARL structure, This would also help us correlate the
> test names with web resources like
> <http://www.w3.org/2001/sw/DataAccess/tests/#testrdfSemantics-001>
>
> Attached is the output of a run. I note the turtle serializer problems
> at the bottom and will look at them later. The stuff at the top may be
> enough to give you and idea what the data looked like.
Fixed the N3Serializer. See new (and pleasantly readable) N3.
--
-eric
home-office: +1.617.395.1213 (usually 900-2300 CET)
+33.1.45.35.62.14
cell: +33.6.73.84.87.26
(eric@w3.org)
Feel free to forward this message to any list for any purpose other than
email address distribution.
Attachments
- text/plain attachment: rdfSemantics-earl.ttl
- image/png attachment: rdfSemantics-earl-sample.png
Received on Monday, 2 October 2006 13:02:28 UTC