Re: Common interface for automatic evaluation tools

On Mon, 8 Sep 2003 22:17:11 -0400 (EDT)
Charles McCathieNevile <charles@w3.org> said:
> 
> This would be a neat trick for developers working in Java - you would be able
> to ship around code. But that might not be relevant to commercial develpments
> where people are using other languages...

This is not a minor problem. Even single manufacturers use several
programming languages in different tools they produce. Sometime
several languages within the same tool.

Besides this, there is also the problem that each test object should
use APIs for accessing the pages (the DOM is ok, but there might be
other libraries needed: for example LIFT is based on the official DOM
enriched with another library called 'semantic analyzer' whose purpose
is to make the tests much more focussed and distinguish a spacer image
from a spacer image used for a hidden button, for example).

In addition each test has to produce some result, and these results
need to handled by the rest of the application. Another API is needed,
but one that is very tightly related with the infrastructure in which
the tool is embedded. For example the results are handled by LIFT for
Dreamweaver (product A) differently than by LIFT for FrontPage
(product B), and very differently from LIFT Machine (product C).

> An interesting part of this would be looking at using Web Services -
> since you ship the document and get a result back you don't really care if
> the code is compatible with your platform or not because you're not running
> it locallly.
> 
> Another possibility would be to provide a semantically rich code description
> for testing functions - inputs, outputs, code language (some good stuff can
> be done in sed, other stuff might be avialable in Java, ...), whether repair
> code is available, etc.
> 
> This would be something like the Testing Description Language that was
> discussed at the EARL workshop in June 2002 for which there were no available
> resources.
> 
> The reason I currently prefer EARL as a general transport is that it is
> simpler. You only have to agree on the test suite being used (in
> accessibility that can be pretty easily done by using external tests and a
> way of namingor pointing to things (which can be done using URIs and Xpath
> for XML content, although there isn't much that is easy otherwise -
> line/character offset hhas the problem that different tools format code
> differently, breaking all these (very brittle) references. If you want to
> agree on shipping test code you need a lot more tight ompatibility design.
> 
> It is possible to write tests for LIFT for Dreamweaver (in Javascript),
> AccVerify (through the interface) or WAINu (as Java code). Making them
> portable is a major hassle. It would be nice if it happened, but seems in
> general to be "too hard" - there are too many reasons why it won't work, and
> finding a way around it by integrating results from different tools seems
> much more achieveable.

I agree. 

        Giorgio Brajnik
______________________________________________________________________
Dip. di Matematica e Informatica   | voice: +39 (0432) 55.8445
Universita` di Udine               | fax:   +39 (0432) 55.8499
Via delle Scienze, 206             | email: giorgio@dimi.uniud.it
Loc. Rizzi -- 33100 Udine -- ITALY | http://www.dimi.uniud.it/~giorgio

Received on Tuesday, 9 September 2003 12:24:23 UTC