RE: potential misuse of warnings (was Re: [ACTION] Warnings in EARL)

Hi Shadi,

See comments inline:

> As per my action item from this week's teleconference 
> discussion [1], please find a scenario for potential misuse 
> of the warnings as proposed by CarlosI [2]:
> 
> [1] <http://www.w3.org/2006/12/06-er-minutes#item04>
> [2] <http://lists.w3.org/Archives/Public/public-wai-ert/2006Dec/0003>
> 
> The scenario proposed by CarlosI highlights the following 
> three separate
> tests:
> 
> * CSS document validity
> * background/foreground color contrast
> * redefinition of properties
> 
> If the question is "does the document validate to CSS" then 
> the answer is yes, and the RDF as presented by CarlosI seems 
> sane (the two latter tests are only informative). However, if 
> the question is "are there accessibility issues in the CSS" 
> then the answer is no because the second test is actually a 
> potential accessibility issue.


First of all I would like to remark that this is IMO the key question. Warnings should be only used as ADDITIONAL and INFORMATIVE messages (The kind of "You could do it better if..." thing), any other use of warning I think it's a missuse.

I think the scenario you're propossing is a CLEAR missuse of accessibility checks. We can provide examples, test cases, test suites and whatever additional clarification, but we can't do anything to stop those who want to clearly missuse the checks.


> So the usage of the warning really becomes dependent on the 
> context of the question and more importantly on the 
> understanding of the tool developers.


It shouldn't be this way, but I agree with you in the fact that it's.


> This means that a 
> potential misuse of the warnings could be as follows:
> 
> <earl:TestResult rdf:ID="result">
>    <earl:validity rdf:resource="&earl;#pass"/>
>    <dc:title xml:lang="en">WCAG AA</dc:title>
>    <dc:description rdf:parseType="Literal" xml:lang="en">
>  ...
>
> Such a misuse is quite likely as some tool developers may 
> prefer to show "pass" to their users rather than "failed" or 
> "can not tell"; and the warning class as is would give them 
> an easy opportunity to do so.


I understand your concerns, but again I think it's a clear missuse.
 

> As a solution I propose the following:
> 
> 1- If a warning is truly informative, then it is only 
> additional information to the result. Thus it does not need 
> additional pointers to a different location and it should 
> only be a simple string.


Even if it's truly informative it may need additional pointers (The original CSS Validator example is a clear one)
As I said I'm against the "warning concept" as a hole and I think it's a bad practice, for example there are not such warnings in TAW (all results map only to one of the EARL validity levels)

Anyway, I don't see any benefit in providing a separated plain text container (not just the dc:description) just to keep people happy. If we follow the warnings path (I personally prefer not to follow it) we should provide a "complete" solution that at least is useful for something (warning locations).


> 2- Other types of more substantial warnings could be done by 
> relating test results to each other. We've had discussion 
> about this in the past and I am aware it is a can of worms 
> but it may make sense.
> 
> Here is a concrete example of what I have in mind: 
> 
> <earl:TestResult rdf:ID="result">
>    <earl:validity rdf:resource="&earl;#pass"/>
>    <dc:title xml:lang="en">LONGDESC Supplied</dc:title>
>  ...
>
> As mentioned, the problem with the "relatedResult" approach 
> is that it is in turn a hidden test description formalism. 
> This in turn reminds me of the "evidence" and "methodology" 
> discussions and not wanting to revive these. Any suggestions 
> to bypass this issue?

If I get the idea correctly, we have a flag (earl:warning) that says "Hi, I'm a Test result that contains warnings.") and then you have a related result about the warning. If we make a few changes in the example you proposed as follows...

<earl:TestResult rdf:ID="result">
   <earl:validity rdf:resource="&earl;#pass"/>
   <dc:title xml:lang="en">WCAG AA</dc:title>
   <dc:description rdf:parseType="Literal" xml:lang="en">
     <div xmlns="http://www.w3.org/1999/xhtml">
       <p>This document passed WCAG AA!</p>
     </div>
   </dc:description>
   <earl:instance rdf:resource="#instance1"/>
   <earl:instance rdf:resource="#instance2"/>
   <earl:warning rdf:type="Literal">Manual evaluation needed to check ...</earl:warning>
   <earl:relatedResult rdf:resource="#result2"/>
</earl:TestResult>

<earl:TestResult rdf:ID="result2">
   <earl:validity rdf:resource="&earl;#cannotTell"/>
   <dc:title xml:lang="en">Check alternatives</dc:title>
   <dc:description rdf:parseType="Literal" xml:lang="en">
     <div xmlns="http://www.w3.org/1999/xhtml">
       <p>Manual evaluation of the alternatives needed.</p>
     </div>
   </dc:description>
   <earl:instance rdf:resource="#instance1"/>
   <earl:instance rdf:resource="#instance2"/>
   <earl:instance rdf:resource="#instanceN"/>
</earl:TestResult>

... Then we have exactly the same problem you wanted to avoid, not to mention that we'll be again in the "evidence/methodology" loop.

My conclusion: if anybody want to do a missuse, they are going to do it.

Regards,
 CI.

 
--------------------------------------

Carlos Iglesias

CTIC Foundation
Science and Technology Park of Gijón
33203 - Gijón, Asturias, Spain 

phone: +34 984291212
fax: +34 984390612
email: carlos.iglesias@fundacionctic.org
URL: http://www.fundacionctic.org

Received on Tuesday, 12 December 2006 16:14:00 UTC