Re: Preliminary results from ER Test Files And Tool

Wendy,

Thanks for the quick feedback. In answer to your questions:

> 1. If the tool being evaluated identifies the problem, but doesn't offer
> the suggestion, does it pass?  For example, test file 1-1-1-f12 is a
> horizontal rule image without alt-text.  InSight correctly identifies that
> it is missing alt-text but doesn't suggest "horizontal rule."  The ATR
tool
> says, "Should trigger:  Image missing ALT text (suggest "horizontal
rule").
>
I think that they have failed this test. But this test comes from the AERT
section on "suggested repairs" so the test doesn't have the weight of some
the other tests. You must mark the tests correctly then process the EARL to
find out how the tool performed. For example, they may have passed only 50
tests but they were all P1 tests so the tool is excellent on P1 items.

> 2. Several of the test files do not say what should trigger.  e.g.,
> 1-1-10-f1, 1-1-12-f1, 1-1-14-f1, 1-1-14-f2, 1-1-14-f3, 1-1-14-f4
>
> 1-1-10-f1.xml  has a comment element while 1-1-1-f1.xml has a
> suggested_message and an empty comment element.
>
Sorry about that

We're still working on the test files and should have this fixed up in the
next couple of days.

> 3.  During my review I made comments about additional messages that
InSight
> was generating as I went along.  Those comments are not recorded in the
> results.  Also, the comment field did not clear when I selected a new test
> file (as I had expected it to do).  To write something new I had to delete
> the text then write.  BUT none of that was captured.
>
Yes, comments are not getting stored in the output file. I'm not exactly
sure how we should store them in EARL and am waiting to hear back from Sean.
I hope to have this implemented very shortly (perhaps tomorrow).

Cheers,
Chris


----- Original Message -----
From: "Wendy A Chisholm" <wendy@w3.org>
To: "Chris Ridpath" <chris.ridpath@utoronto.ca>; "WAI ER IG List"
<w3c-wai-er-ig@w3.org>; <josh@zafu.com>
Sent: Thursday, August 02, 2001 11:14 AM
Subject: Preliminary results from ER Test Files And Tool


> Chris and Josh,
>
> This is great!  I've begun a preliminary review of SSB Technologies
InSight
> (version 2.4 - couldn't download the latest - I'll have to talk with Tim
> again).  The results that I have so far are at:
> http://www.w3.org/WAI/ER/2001/07/insight.rdf
>
> In the quick 10 tests that I did, I found some interesting errors in
InSight.
>
> I have a few questions/suggestions about the ATR tool and test files.
>
> 1. If the tool being evaluated identifies the problem, but doesn't offer
> the suggestion, does it pass?  For example, test file 1-1-1-f12 is a
> horizontal rule image without alt-text.  InSight correctly identifies that
> it is missing alt-text but doesn't suggest "horizontal rule."  The ATR
tool
> says, "Should trigger:  Image missing ALT text (suggest "horizontal
rule").
>
> I said that it passed it, since it did trigger.  Perhaps these ought to be
> separate - the suggestion from the error detection.  I think it would be
> good to know if the correct suggestion was made or if any suggestion was
made.
>
>
> 2. Several of the test files do not say what should trigger.  e.g.,
> 1-1-10-f1, 1-1-12-f1, 1-1-14-f1, 1-1-14-f2, 1-1-14-f3, 1-1-14-f4
>
> 1-1-10-f1.xml  has a comment element while 1-1-1-f1.xml has a
> suggested_message and an empty comment element.
>
>
> 3.  During my review I made comments about additional messages that
InSight
> was generating as I went along.  Those comments are not recorded in the
> results.  Also, the comment field did not clear when I selected a new test
> file (as I had expected it to do).  To write something new I had to delete
> the text then write.  BUT none of that was captured.
>
>
> Thanks for your work Chris and Josh!
> --wendy
> --
> wendy a chisholm
> world wide web consortium
> web accessibility initiative
> seattle, wa usa
> /--

Received on Thursday, 2 August 2001 11:53:21 UTC