Re: Test Framework Task Force

On 20/03/2013 23:47, Tobie Langel wrote:
>> 5) the framework test harness records the evaluator (person) name and
>> >produce reports by evaluator
> That shouldn't be an issue either, except perhaps in terms of the UI for it.

I also requested this some time ago. Often you find results recorded 
that are not expected. It would be very useful to be able to contact the 
person who submitted the results to check whether this was a mistake or, 
if not, further investigate what happened.  IP addresses are only 
partially useful for spotting certain patterns, but not much use beyond 
that.

>> >6) the framework test harness allows entry of text comments by the evaluator
> It's not clear to me how this would work. Would this be a comment per test multiple comments per tests, or for the overall test suite? Would these comments be stored with the test results, or would they be displayed on every subsequent test run?

Another thing I'd also like to see, since we sometimes need it for i18n 
tests too. For example, sometimes it's just one aspect of a test that 
fails, and it would be good to capture information about that - for an 
example see the notes below the table at 
http://www.w3.org/International/tests/html-css/list-style-type/results-list-style-type#alphabetic 
which is testing that the list-style-type property is correctly applied 
across a range of numbers.

This would also give testers a chance to report tests they think are 
broken or invalid.

So, in my case, I'm looking for something on the page that displays the 
Pass, Fail, etc buttons that says something like "Leave a comment on 
this test".

hope that helps
RI




-- 
Richard Ishida, W3C
http://rishida.net/

Received on Thursday, 21 March 2013 11:52:31 UTC