RE: more discussion on warnings in EARL

Hi Shadi, everybody. 
Comments below.

> I'm wondering whether the concept of "warning" (as we have 
> been discussing it) really belongs in EARL. During the last 
> face-to-face we had decided that EARL only records the 
> *outcome* of tests and not additional information about the 
> relationship of the tests. However, issuing a warning seems 
> to require knowledge about how the tests relate.
> 
> For simple and purely informative warnings (such as "LONGDESC 
> not supported by many browsers"), it may be handy to record 
> this directly in EARL. However, as we get to more complex 
> types of warnings (such as the CSS validation example in 
> [1]), it starts to get into the application level and 
> somewhat messy. Below are two types of warning to outline how 
> these could be addressed outside EARL:


I agree with you on the overall approach, but I think that we could still have problems in same use cases (see next comments about mobileOK example)

 
> #1. WARNINGS THROUGH TESTING WORKFLOW
> <http://www.w3.org/TR/2006/WD-mobileOK-basic10-tests-20061113/
> #test_auto_refresh>
> - The testing framework, in this example the draft MobilOK 
> Test 3.1, defines a clear sequence for carrying out four 
> atomic tests. Each result for each of these four tests could 
> be recorded in an EARL assertion. The overall outcome of Test 
> 3.1 depends on how these four atomic tests are combined. Also 
> when a warning is issued is defined by the framework.
> - In this case, it doesn't make sense for the tool to issue a 
> warning when the test "If a Refresh HTTP header is present" 
> is positive. The warning may or may not be issued depending 
> on the outcome of the next test to be carried out. This also 
> applies to the CSS examples brought up in earlier threads on 
> the topic of warnings in EARL.
> - Conclusion: the warning in this case is not on the level of 
> EARL but comes from the higher level testing framework.


Maybe I'm missunderstanding the mobileOK test cases, but I think that there are several atomic test that specify an WARN output as test validity level in a explicit way. The question then is, if we want to record the results of these atomic tests how are we going to record them?

For example: If the document's Internet media type is "text/html", WARN (not pass nor fail). How does this atomic test Test Result look like?

Anyway, and if I've correctly understood what you're thinking about, an example of what you mean could be done again with the CSS example:

<earl:TestResult rdf:ID="result1">
  <earl:validity rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#pass"/>
  <dc:title xml:lang="en">W3C CSS Validator Results</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>This document is valid CSS</p>
    </div>
  </dc:description>
</earl:TestResult>

<earl:TestResult rdf:ID="result2">
  <earl:validity rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail"/>
  <dc:title xml:lang="en">Same color and background-color</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>Same colors for color and background-color in two contexts</p>
    </div>
  </dc:description>
  <earl:instance rdf:resource"#instance1"/>
</earl:TestResult>

<earl:TestResult rdf:ID="result3">
  <earl:validity rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail"/>
  <dc:title xml:lang="en">No generic font family found</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>You are encouraged to offer a genric family as a last alternative</p>
    </div>
  </dc:description>
  <earl:instance rdf:resource"#instance2"/>
  <earl:instance rdf:resource="#instance3"/>
  <earl:instance rdf:resource="#instanceN"/>
</earl:TestResult>

<earl:PointerCollection rdf:about="#instance1">
  <earl:LineCharLenPointer>
    <earl:line>55</earl:line>
    <earl:char>12</earl:char>
  </earl:LineCharLenPointer>
</earl:PointerCollection>

<earl:PointerCollection rdf:about="#instance2">
  <earl:LineCharLenPointer>
    <earl:line>145</earl:line>
    <earl:char>1</earl:char>
  </earl:LineCharLenPointer>
</earl:PointerCollection>

<earl:PointerCollection rdf:about="#instance3">
  <earl:LineCharLenPointer>
    <earl:line>154</earl:line>
    <earl:char>1</earl:char>
  </earl:LineCharLenPointer>
</earl:PointerCollection>

<earl:PointerCollection rdf:about="#instanceN">
  <earl:LineCharLenPointer>
    <earl:line>193</earl:line>
    <earl:char>1</earl:char>
  </earl:LineCharLenPointer>
</earl:PointerCollection>

We record just results, all of them come from testcases that could determine whether they are warnings or not. There is no distinction at the result level.
 

> #2. WARNINGS THROUGH TEST DESCRIPTIONS
> <http://www.w3.org/TR/2006/WD-WCAG20-TECHS-20060427/#H45>
> - The test description, in this example the WCAG 2.0 
> Technique H45, describes the applicability of the test. For 
> example that "Some older assistive technologies do not 
> support the longdesc attribute. Supported in leading 
> Assistive technologies now." This type of information may be 
> important for the developer and/or evaluator and may need to 
> be communicated in a different way.
> - In this case, the tool could issue a warning as soon as it 
> carries out this test. The warning is purely informative and 
> supplements the outcome of the test. However, the text of 
> this warning (that the tool issues to the end user) is 
> already in the test description and does not have to be 
> recorded in EARL as well.
> - Conclusion: the warning in this case *could* be stored in 
> EARL but it should also come from the higher level test description.


Totally agree on this point.


> So again, we currently have the following three options on 
> what to do with warnings in EARL:
> 
> A. drop the idea of warnings in EARL completely
> B. adopt a simple literal earl:warning property
> C. adopt a more comprehensive earl:Warning class


Then I've changed my mind now. I think that the best practices scenarios should just rely on A and maybe B, and we shouldn't provide built in warning capabilities in EARL more than simple literal messages, mainly because I see now they are more a test development question than a result question. Nevertheless, the problem we still have is, how are we going to manage use cases where the testcase itself specifies a WARN output as validity level?


> [1] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Dec/0003>

Regards,
 CI.
 
--------------------------------------

Carlos Iglesias

CTIC Foundation
Science and Technology Park of Gijón
33203 - Gijón, Asturias, Spain 

phone: +34 984291212
fax: +34 984390612
email: carlos.iglesias@fundacionctic.org
URL: http://www.fundacionctic.org

Received on Friday, 15 December 2006 10:08:51 UTC