W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2011

Re: Automatic test results reporting

From: Bryan Sullivan <blsaws@gmail.com>
Date: Thu, 11 Aug 2011 07:23:38 -0700
To: "Linss, Peter" <peter.linss@hp.com>
CC: Francois Daoust <fd@w3.org>, Philippe Le Hegaret <plh@w3.org>, public-test-infra <public-test-infra@w3.org>
Message-ID: <CA6933FA.12F4F%blsaws@gmail.com>
OK, also as Francois suggested having an export button for the test results
would work for submitting in batch. Thus once desired results are obtained,
the tester can upload them.

However it is done, the key objective I have is to enable a reliable,
low-overhead system of test results collection. Automation is preferable to
avoid the overhead and error risk of manual actions. Local collection of
results with a one-click submission at test end (for which even the click is
automatable) should also be an option.

Given varying capabilities of the Web agents under test, there will need to
be multiple methods of results collection supported.

Bryan

On 8/10/11 11:14 AM, "Linss, Peter" <peter.linss@hp.com> wrote:

> The harness already has the capability to import results from a text file,
> this lets users gather results from other testing systems and submit them in a
> batch (it just takes an administrator to do the import currently, adding an
> upload page for users is on the todo list).
> 
> Adding an API to accept results from a different system would be fairly
> trivial, my main concern would be adding some authentication to that API so
> that it can't get spammed. There's a user/login system growing under the hood
> (from common code used with the test suite management system), that could be
> used for that so only registered users can get to the API (or upload batch
> results).
> 
> Peter
> 
> On Aug 10, 2011, at 6:19 AM, Bryan Sullivan wrote:
> 
>> Hi Francois,
>> 
>> A useful feature would be to allow a tester to define the test results
>> collection server address, and for W3C to publish the API that the client
>> side of the framework will use to deliver the results. This will allow
>> testers to do preliminary testing, and automatically collect the results at
>> their own server for offline/private review, prior to doing a test for which
>> results are delivered to W3C.
>> 
>> Selection of a non-default results server could be a configuration item
>> which the tester can set at the start of a test session or persistently (eg
>> via localstorage).
>> 
>> Bryan
>> 
>> On 8/10/11 1:33 AM, "Francois Daoust" <fd@w3.org> wrote:
>> 
>>> Hi Philippe,
>>> 
>>> I've added an automatic test results reporting feature to the framework,
>>> see:
>>>  http://w3c-test.org/framework/suite/nav-timing-default/
>>> 
>>> The page presents a "Report test results automatically (when possible)"
>>> checkbox, enabled by default. When activated, pressing "start" will run the
>>> test suite, jumping for test case to test case and saving the result
>>> reported
>>> by "testharness.js".
>>> 
>>> That's a bit ugly from a pure visual point of view as the browser jumps from
>>> page to page as soon as it gets the result of the test, but we can improve
>>> that later on, I think.
>>> 
>>> Note automatic results reporting gets automatically deactivated when the
>>> user
>>> chooses to report for a user-agent that is not the user-agent on which it
>>> runs
>>> the tests.
>>> 
>>> Francois.
>>> 
>> 
>> 
>> 
> 
Received on Thursday, 11 August 2011 14:24:04 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 11 August 2011 14:24:05 GMT