Re: Add todo test outcome

Hi Kjetil,

Thanks for this, glad to hear that Solid is using EARL!

Indeed, EARL is currently not actively maintained anymore but there is 
no harm logging ideas and discussion.

One of the issues of EARL, is that it was designed with the expectation 
of a "test case description language" counterpart. This would describe 
things like how tests relate to each, the order they should be executed 
in, and the implication of one result on another test. EARL was focused 
on describing results only. That is, after tests are executed.

Here are some references:
  - https://www.w3.org/QA/WG/2003/10/tcdl-20031013.html
  - https://www.w3.org/WAI/ER/tests/usingTCDL
  - http://publica.fraunhofer.de/documents/N-45007.html

It seems the TODO feature belongs in some TCDL rather than in EARL? Or 
maybe the future of EARL should be to include TCDL aspects?

Best,
   Shadi


On 26/07/2021 18:21, Kjetil Kjernsmo wrote:
> Dear all,
> 
> The Solid specification test harness is using EARL for the reports, and I
> noticed we lack a predicate for tests are should be executed, but the
> outcome should not count towards the pass or fail counts. It is very useful
> to have such an outcome when developing things. In particular, we could use
> it to record when a server responds with 501 Not Implemented.
> 
> In fact, such an outcome is part of the Test Anything Protocol, where it is
> known as TODO tests:
> https://testanything.org/tap-version-13-specification.html#todo-tests
> Even though we're not using TAP anymore, I note that TAP is used in 30+
> languages for tests. To be able to record TAP output fully would be great.
> In fact, I once started a rough implementation to do it that:
> https://metacpan.org/pod/TAP::Formatter::EARL
> 
> I realize that EARL is not actively maintained anymore, and so the chances
> to get a test outcome (e.g. a predicate earl:todo) is slim, but if some
> opportunity should arise, I would love to see this go in. :-)
> 
> Cheers,
> 
> Kjetil
> 
> 
> 

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)

Received on Tuesday, 27 July 2021 07:51:46 UTC