Re: Draft ERT WG Charter for review and comment

> we need tools that can incorporate EARL report..
>
Yes. EARL itself needs several tweaks so that reports from one tool are useful to other tools. We also need to tweak it so that reports from various tools can be combined into one document.

> ...idea that was discussed sometime ago was to develop 
> a service that would be able to upload EARL reports 
> produced by several tools and offer to the user a sort of 
> integration and comparison of results.
>
This has been difficult because all of our tools are testing for slightly different things. Let's agree on what we should be testing for.

> Secondly, I would not like AERT WG to focus on 
> technical aspects of what techniques AER tools
> should/could employ to evaluate websites.
> Let the developers do this.
>
I disagree. The guideline authors should tell us what they require to achieve compliance to their guidelines. We should not leave this up to the tool developers. The current situation where all our tools provide different results makes the user very confused and dilutes the guidelines.

> Why not concentrating on understanding how to 
> measure the effectiveness, usability, quality of use, 
> accessibility of tools?
>
Yes, this certainly requires more work.

> The service AERT WG would provide in this case is 
> again towards end-users, purchasers, tools manufacturers, 
> etc in terms of a well-studied and robust tool evaluation 
> methodology, that would be able to appropiately classify
> tools and results produced by tools.
> A first draft of an example of methodology is described...
>
I didn't see the methodology in the attached documents. Could you explain a bit more about it?

Cheers,
Chris




  ----- Original Message ----- 
  From: Giorgio Brajnik 
  To: w3c-wai-er-ig@w3.org 
  Cc: chris.ridpath@utoronto.ca 
  Sent: Monday, August 30, 2004 5:42 AM
  Subject: Re: Draft ERT WG Charter for review and comment


  Hello to everybody.
  I'd like to contribute with my view of what AERT might be focussing
  on. This is more an open discussion rather than specific points to be
  made to the charter proposal. My apologies for this generality.

  I think we need to establish some sort of consensus on a long-term
  vision of what the group is about, and then define some of the steps
  (and hence deliverables) that should be produced in the short-term to
  achieve progress along those lines.

  As for the long-term vision, what is the service that AERT WG is going
  to offer and to whom?

  EARL is not directly an answer, as it is a means to achieve something:
  better interoperability between tools, so that end-users, tool
  manufacturers, possible purchasers, consumer associations, etc could
  be able to compare tools and could be able to compare/integrate
  different evaluations on certain websites. To achieve this we
  need tools that can incorporate EARL report, process them, and produce
  interesting and useful results, not simply tools that can only generate 
  EARL reports.

  I think the AERT WG should keep constantly under the radar how EARL is
  or might be used, and do what can be done to promote its usage. One
  idea that was discussed sometime ago was to develop a service that
  would be able to upload EARL reports produced by several tools and
  offer to the user a sort of integration and comparison of results. The
  service could then be used by the public just like the HTML or CSS 
  validators are used now. I would consider this a great success of AERT WG.

  Secondly, I would not like AERT WG to focus on technical aspects of
  what techniques AER tools should/could employ to evaluate
  websites. Let the developers do this.

  Why not concentrating on understanding how to measure the
  effectiveness, usability, quality of use, accessibility of tools?
  Development of an appropriate methodology could lead to (i) valid and
  useful comparisons between tools, (ii) and corresponding labeling (in
  terms of strength and weaknesses) of evaluations performed by the
  tools (perhaps evaluations could also be compared if available in
  EARL).

  The service AERT WG would provide in this case is again towards
  end-users, purchasers, tools manufacturers, etc in terms of a
  well-studied and robust tool evaluation methodology, that would be
  able to appropiately classify tools and results produced by tools.

  If appropriate, AERT WG could also run evaluations of tools according
  to such a methodology, and "sell" these results.

  A first draft of an example of methodology is described

  Best regards to all of you

           Giorgio Brajnik
  ______________________________________________________________________
  Dip. di Matematica e Informatica   | voice: +39 (0432) 55.8445
  Universita di Udine                | fax:   +39 (0432) 55.8499
  Via delle Scienze, 206             | email: giorgio(at)dimi.uniud.it
  Loc. Rizzi -- 33100 Udine -- ITALY |

  and UsableNet Inc.