W3C home > Mailing lists > Public > public-webapps-testsuite@w3.org > April 2013

Re: How to define test result in report if multiple test points designed in one test file?

From: Arthur Barstow <art.barstow@nokia.com>
Date: Mon, 08 Apr 2013 10:20:33 -0400
Message-ID: <5162D231.2020803@nokia.com>
To: "ext Zhao, Tina" <tina.zhao@intel.com>
CC: "public-webapps-testsuite@w3.org" <public-webapps-testsuite@w3.org>
On 4/3/13 3:17 AM, ext Zhao, Tina wrote:
>
> The draft the Opera's SSE test report is available at 
> http://www.w3.org/wiki/Webapps/Interop/ServerSentEvents
>
> I have a question about test result measurement:
>
> Take [1] test file for example. There are four check points in one 
> file, only if all of them get pass, the test result shows PASS (other 
> ways Fail) in report, right? However, in traditional QA way, I thought 
> this should be four test cases, 3 PASS, 1 NotRun (so I highlighted the 
> details in report).
>
> Do we have rule or guideline for this?
>

No, we don't have any strict rules regarding test result reporting.

If a row/line in the test report is a summary of more than one test, 
then if any of that row's tests fail, I think the Principle of Least 
Surprise suggests an overall status of Fail makes sense.

It is also acceptable to report the results such that every test has its 
own row/line (even if there are multiple tests per file). Presumably 
that would be more work for you, but AFAIC, that would be fine. (This is 
what Lachlan did with the Selectors API Implementation Report 
<http://dev.w3.org/2006/webapi/selectors-api-testsuite/level1-baseline-report.html>.)

-Art


> [1] 
> https://github.com/w3c/event-source/blob/master/eventsource-constructor-stringify.htm
>
> Thanks,
>
> Tina
>
Received on Monday, 8 April 2013 14:21:06 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 18:52:57 UTC