Re: Draft ERT WG Charter for review and comment

On Mon, 30 Aug 2004, Chris Ridpath wrote:

>> we need tools that can incorporate EARL report..
>Yes. EARL itself needs several tweaks so that reports from one tool are
>useful to other tools. We also need to tweak it so that reports from various
>tools can be combined into one document.

>> ...idea that was discussed sometime ago was to develop
>> a service that would be able to upload EARL reports
>> produced by several tools and offer to the user a sort of
>> integration and comparison of results.
>This has been difficult because all of our tools are testing for slightly
>different things. Let's agree on what we should be testing for.

There are two approaches. EIther we all agree to run the same tests (in which
case we are essentially saying that this is only useful for the Lowest Common
Denominator, since it won't cover anything else) or we describe how to map
between differnt tests. I think the latter is a much more useful approach, in
part because it allows me to take results from two different tests and use
them together, taking advantage of the fact taht different companies test
different things better or worse (when given complete freedom to design their
own tests).

>> Secondly, I would not like AERT WG to focus on
>> technical aspects of what techniques AER tools
>> should/could employ to evaluate websites.
>> Let the developers do this.
>I disagree. The guideline authors should tell us what they require to
>achieve compliance to their guidelines. We should not leave this up to the
>tool developers. The current situation where all our tools provide different
>results makes the user very confused and dilutes the guidelines.

I agree strongly with Giorgio. The guidelines should tell us what they need.
However given that many of the necessary tests require human judgement, and
that the guidelines often don't tell us what they need (HTML conformance,
WCAG 1 conformance, CSS conformance and many other kinds of guidelines where
EARL is a useful conformance reporting format) I think it is unrealistic to
rely on this. EARL was designed in RDF rather than XML for precisely this

That is not to say that developers should not ask the WCAG grup to confirm
whether or not their test is accurate. Just to note that at the moment there
is no reliable way of doing this (especially for developers not working in

>> Why not concentrating on understanding how to
>> measure the effectiveness, usability, quality of use,
>> accessibility of tools?
>Yes, this certainly requires more work.
>> The service AERT WG would provide in this case is
>> again towards end-users, purchasers, tools manufacturers,
>> etc in terms of a well-studied and robust tool evaluation
>> methodology, that would be able to appropiately classify
>> tools and results produced by tools.
>> A first draft of an example of methodology is described...
>I didn't see the methodology in the attached documents. Could you explain a bit more about it?

Euroaccessibility used to have a group that worked on that. But the group,
like ERT, expired. There is no evidence that anyone is talking about
resurrecting the EuroAccessibility group.



Received on Sunday, 5 September 2004 17:32:52 UTC