Re: Formalizing testSubject

At 06:34 PM 2002-03-02 , Sean B. Palmer wrote:
>In EARL, the TestSubject is the thing being evaluated. There are two
>requirements for identifying test subjects in EARL: a) we must be able
>to refer to anything, and b) we must be very clear about what is being
>identified. 

There is no way to satisfy both a) and b) as stated.  Lots of the things that one would like to be able to refer to under a) posess a certain degree of ineffability, such that b) is impossible.  Fortunately, the requirement as stated is overly strong and there are practical solutions for practical applications of EARL.  

The actual requirement for evaluation transactions is to preserve enough information about what was evaluated so that another party can reproduce your results.  In real life situations with imperfect persistence and transaction systems, this includes everything the supplier of the resource has designated as identifying marks in their publishing transaction, plus more.

The QA group should be consulted on this point.  They have configuration-identification spaces to deal with that they will know about.

>Examples of things that we may want to evaluate in EARL
>are:-
>
>   * A web page http://example.org/ on a particular date

To retain the possibility of diagnosis, one needs to be more skeptical of the URI -to- test inputs relationship.

For a web page one wants to retain:
all headers in the HTTP transmission by which received.  The precedence order for use in queries would be something like
- the Location: value from the original request
- the Location: value from the actual response that eventually returned an entity body [in HTTP terms] after all redirects.
- the Last-Change-Date: timestamp if provided
- the timestamp of the HTTP transmission itself
- others as required.

Then, in terms of the actual data what one wants to retain depends on the test outcome.  But it will either be an entire verbatim copy of the resource representation as received in the HTTP transmission or some hash or checksum derived from it that will almost surely detect if different evaluators had different data as representations of the same resource.


Not a precise enough identification to isolate situations where different evaluators reach different conclusions because they obtained different [concrete forms] of a resource under the same URI and timestamp conditions.

Real systems, by which I mean something comprising enough stuff to encompass whole bugfix loops, which use EARL, will need some provisions for resolving these situations.  The bane of test programs is the "re-test OK" case, where a problem is encountered but the conditions that created the problem are not captured well enough to reproduce the problem in a controlled environment where traceback and diagnosis of the root cause is possible.

Test systems have to be skeptical about the reproduceability of what you get from accessing a nominally same resource.  The basic rules should be on test complete pass save some checksum for the actual thing evaluated; and on test yielding a discrepancy save the whole thing.  Until the maintainer of the thing evaluated has corroborated the problem and accepted the trouble report, all evidence should be presumed needed for further diagnostic analysis.

If there is a problem, you cannot separate the 'resource' from the service process that delivers different users represenations of the 'resource' until the source of the problem has been found.  Any and all details of the actual data or transformation as it existed in the concrete test transaction are of interest for diagnostic purposes.

If the supplier of the web page or tool has computed a Content-MD5: value this should be verified before discarding the concrete test subject, but it may be used as the associated hash for precise resource representation matching.  If one is evaluating a tool, and the tool supplier gives identifying codes for the product (such as media serial number) these and any verified digital signatures are also usable.

Or if the testing activity wants to save data storage and not retain the evidence, then the testing activity should compute a hash of their own chosing that will be highly resistant to false compares between different evaluators of the same test subject.

>   * A part of a web page - just an element, or a few elements

This should be done by a resource identification plus an index into the representation which is reproduceable within the class of representations of the same type, as in URI plus XPointer.



>   * An accessibility tool, or a User Agent

Here the configuration identification used by the supplier, perhaps augmented by a hash to double check for data control problems in the distribution channels, is what to use.

At 06:44 PM 2002-03-02 , Sean B. Palmer wrote:
>>    a) Use testSubject and not reprOf/documentation.
>>    b) Let people choose: testSubject or reprOf/documentation
>>    c) Use reprOf/documentation and not testSubject
>>
>> Option a will depend upon people including an rdf:type arc,
>> option b may do if they choose to use testSubject, and the
>> latter option will not rely on rdf:type at all since the domains
>> will take care of it.
>
>Oops - something I didn't mention: b) will basically be the most
>complicated to query, followed by c), and with a) as the easiest. It's
>annoying that the "broken" option is also the easiest to query. Of
>course, if we want to go with it, we can just draft up some prose
>saying "hey, you've got to put an rdf:type property arc on your
>TestSubject, and it has to tell us what kind of test subject it is
>w.r.t. evaluating".

What kind of a thing it is, as contemplated in the evaluating, sounds like a property of the evaluation applied, more than a property of the thing evaluated.

To put it another way, unless the view or aspect of the identified resource that is being evaluated is built into the release package as contemplated and identified by the dc:creator of the thing evaluated, this projection or view transform should be handled as information about the test applied and not as identification of the subject to which the evaluation was applied.

The default page one is served on issuing a GET against http://www.example.com/ can be evaluated as an accessible web page within its own four walls or evaluated as the hub of an organized set of web pages.  These are different aspects each with its own variant criteria.  But the URI and even the bits are the same.

For identifying the thing evaluated it is best to stick to properties and characterizations that would be recognized as identifying properties proper to that thing by the dc:creator of the thing in question.  


A car [in the U.S.] has a Vehicle Identification Number (VIN) allocated and identifying marks bearing this number affixed to the car by the manufacturer.  The manufacturer understands that it will be viewed as a member of the class of products that requires such a number, and will be indexed or identified in licensing processes and so forth by this number.

On the other hand, if you decide to evaluate this car as a possible police car, or possible taxicab, this construction of the terms of engagement is introduced by you and not by the manufacturer.  To identify the car evaluated, you just use the VIN, not VIN+police_car or VIN+taxicab.

Al



>
>This much we already know. In EARL 0.95, sometimes we have a necessary
>level of indirection. For example, if you want to point to the page
>http://example.org/ as it was on 2002-03-02, you can't just do
>something like:-
>
>   <http://example.org/> earl:fails :SomeTest; earl:date "2002-03-02"
>.
>
>because you'll lose the date information on merging. So instead, we
>create a new thing - bNoded, or with a URI - something like [
>earl:testSubject <http://example.org/>; earl:date "2002-03-02" ] .
>This can be read as "a thing which is the web content at
>http://example.org/ on the date 2002-03-02". In fact, there is some
>information left out there. We don't really know that we're evaluating
>a bit of WebContent at all...
>
>   [ a earl:WebContent; earl:date "2002-03-02";
>      earl:testSubject <http://example.org/tool> ] .
>   [ a earl:Tool; earl:date "2002-03-02";
>      earl:testSubject <http://example.org/tool> ] .
>
>In this case, it's clear(er) what's being evaluated. In the top
>example, we're evaluating the documentation for the tool, and in the
>bottom example we're evaluating the tool documented at
>http://example.org/tool. Unfortunately, there's no way that we can
>either a) force people to use a type arc, or b) deduce what the author
>of the URI *means* by it, unless they have stated that at the resource
>itself.
>
>So there's a problem there - ambiguity. I previously resolved [1] to
>clear this up by introducing two new properties that would take the
>place of testSubject: earl:reprOf, and earl:documentation. For example
>(now assuming an EARL 1.0 namespace):-
>
>   [ earl:reprOf <http://example.org>; earl:date "2002-03-02" ] .
>   [ earl:documentation <http://example.org>; earl:date "2002-03-02" ]
>.
>
>rdf:type arcs aren't strictly necessary here since we know that the
>domain of earl:reprOf is earl:WebContent, and that the domain of
>earl:documentation is a union of earl:Tool and earl:UserAgent.
>
>But it isn't necessary that we deprecate testSubject if people want to
>include it; we can rely on people using rdf:type to tell us the kind
>of content (as we do in EARL 0.95). A schema snippet for this is at
>[2].
>
>So there's a choice for EARL 1.0:-
>
>   a) Use testSubject and not reprOf/documentation.
>   b) Let people choose: testSubject or reprOf/documentation
>   c) Use reprOf/documentation and not testSubject
>
>Option a will depend upon people including an rdf:type arc, option b
>may do if they choose to use testSubject, and the latter option will
>not rely on rdf:type at all since the domains will take care of it.
>

>There are also some issues about identifying bits of representations
>etc., but that's not strictly a part of this debate, IMO.
>
>Cheers,
>
>[1] http://lists.w3.org/Archives/Public/w3c-wai-er-ig/2001Dec/0018
>[2] earl:TestSubject rdfs:subClassOf
>        [ daml:onProperty earl:testSubject; daml:cardinality "1" ] .
>   earl:reprOf rdfs:subPropertyOf earl:testSubject .
>   earl:documentation rdfs:subProperty earl:testSubject .
>
>--
>Kindest Regards,
>Sean B. Palmer
>@prefix : <http://purl.org/net/swn#> .
>:Sean :homepage <http://purl.org/net/sbp/> .
>

Received on Sunday, 3 March 2002 08:18:43 UTC