Test Anything Protocol (TAP)

Hi,

I have an action item [1] to look into the Test Anything Protocol (TAP)
TAP has been part of Perl since 1987 (i.e. the first version of Perl) 
and has been implemented in other languages, for example C [2], Java 
[3] and PHP web frameworks such as Symphony [4]. In spite of this, 
there is no formal specification for the protocol; the behaviour of 
Perl's Test::Harness module is the de-facto TAP standard, along with 
a writeup of the specification on CPAN [5].
The CPAN documentation describes TAP as a "simple text-based 
interface between testing modules such as Test::More and the test 
harness Test::Harness."
It is "a protocol to allow communication between unit tests and a 
test harness. It allows individual unit tests (TAP producers) to 
communicate test results to the testing harness in a 
language-agnostic manner. Tests can indicate success or failure, mark 
tests as unimplemented or skipped, and provide additional information 
to help debug failed tests. Unlike other testing systems, it can also 
indicate missing tests and duplicated tests." [6]

An example test session may look like [7]:

1..9
  ok 1 - Beverage::Coffee loaded.
  ok 2 - Cup creation.
  ok 3 - Kettle location.
  ok 4 - Boiling water.
  ok 5 - Black coffee.
  not ok 6 - Milk location.
  # Milk not found in fridge.
  ok 7 - White coffee. # Skipped, no milk.
  ok 8 - Espresso # Skipped, espresso machine not available.
  not ok 9 - Non-dairy creamer # TODO, unimplemented.

The line "1..9" tells the test suite to expect 9 tests. The first 8 
tests pass, the last one fails.
My current understanding is that this can be logged, so there is no 
XML or RDF-based report format, unless the TAP consumer converts the 
output to something else. A prosal for an output format is YAMLish, a 
subset of YAML [8] (the same page also discusses JSON).


With regard to Sean B. Palmer's questions in Bug 031 [101]:

Q: Does EARL replicate their efforts?
A: Based on this first brief look, I don't think so.

Q: Do their efforts replicate EARL?
A: Same as previous question.

Q: Why do TAP and xUnit XML have things that EARL doesn't?
A: Which things does TAP have that EARL doesn't? TAP is a protocol, 
EARL is a report format. It seems that a TAP consumer could be 
written that produces EARL.
There may be a few things in TAP that we don't have in EARL, for 
example TAP knows something called "Bail out!", i.e. when a test is 
aborted unexpectedly and the TAP consumer should ignore the rest of 
the TAP syntax [102]. Is this something we should add to EARL or that 
should be left to other vocabularies or extensions?
TAP also knows comments [103], which can be used for "diagnostics" 
[104]. I think this can be mapped to the info property  [105].


Q: Has there been any contact between the groups of people working on 
each of these three?
A: ?

Q: Are there other languages which may have a similar bearing on 
EARL? Why wasn't this covered in the Requirements for the Evaluation 
and Report Language (EARL) 1.0 document?
A: ?



[1] <http://www.w3.org/2011/09/07-er-minutes.html#action02>
[2] <http://www.eyrie.org/~eagle/software/c-tap-harness/writing.html>
[3] tap4j: <http://tap4j.org/junit_support.html>
[4] A Gentle Introduction to symfony: Chapter 15 - Unit And 
Functional Testing: 
<http://www.symfony-project.org/gentle-introduction/1_4/en/15-Unit-and-Functional-Testing>
[5] Test::Harness::TAP : 
<http://search.cpan.org/~petdance/Test-Harness-2.64/lib/Test/Harness/TAP.pod>
[6] 
<http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#Abstract>
[7] <https://www.socialtext.net/perl5/testing#tap_test_anything_protocol>
[8] <http://testanything.org/wiki/index.php/YAMLish>

[101] Bug 031: Relationship to TAP and xUnit XML: 
<http://lists.w3.org/Archives/Public/public-earl10-comments/2011May/0032.html>
[102] 
<http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#Bail_Out>
[103] 
<http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#Comments>
[104] see for example 
<http://search.cpan.org/~petdance/Test-Harness-2.64/lib/Test/Harness/TAP.pod#Diagnostics>
[105] info Property: <http://www.w3.org/TR/EARL10/#info>

Best regards,

Christophe


-- 
Christophe Strobbe
K.U.Leuven - Dept. of Electrical Engineering - SCD
Research Group on Document Architectures
Kasteelpark Arenberg 10 bus 2442
B-3001 Leuven-Heverlee
BELGIUM
tel: +32 16 32 85 51
http://www.docarch.be/
Twitter: @RabelaisA11y
---
Open source for accessibility: results from the AEGIS project 
www.aegis-project.eu
---
Please don't invite me to Facebook, Quechup or other "social 
networks". You may have agreed to their "privacy policy", but I haven't.

Received on Wednesday, 14 September 2011 13:56:29 UTC