Re: Automatic test results reporting

On Aug 10, 2011, at 9:49 AM, Francois Daoust wrote:

> On 08/10/2011 01:51 PM, James Graham wrote:
>> On 08/10/2011 10:33 AM, Francois Daoust wrote:
>>> Hi Philippe,
>>> 
>>> I've added an automatic test results reporting feature to the framework,
>>> see:
>>> http://w3c-test.org/framework/suite/nav-timing-default/
>>> 
>>> The page presents a "Report test results automatically (when possible)"
>>> checkbox, enabled by default. When activated, pressing "start" will run
>>> the test suite, jumping for test case to test case and saving the result
>>> reported by "testharness.js".
>>> 
>>> That's a bit ugly from a pure visual point of view as the browser jumps
>>> from page to page as soon as it gets the result of the test, but we can
>>> improve that later on, I think.
>>> 
>>> Note automatic results reporting gets automatically deactivated when the
>>> user chooses to report for a user-agent that is not the user-agent on
>>> which it runs the tests.
>> 
>> Auto-submitting information to a third-party is an incredibly bad default. I skimmed this email, didn't quite appreciate what the feature did and was quite surprised when the harness said it had reported the results to the W3C. Happily I didn't run it on some secret internal build that happened to have new features covered by the tests, but I could have done.
> 
> True. I switched off the feature by default. I still need to switch form submission to use "post" to prevent the "auto" setting from showing up in URLs.
> 
> I also updated the front page to make it clearer that the user is about to enter test data that will be published online. That includes replacing the "start" button with "Submit test results", completing the checkbox message and adding a note on collected/published info.
> 
> I think your comment extends to more than just automatic test result gathering in any case. The interface is unclear right now on the info that gets collected and published for manual tests as well. That needs updating as well.
> 
> 
>> When the test finishes I would expect it to a) present the results to the user and maybe b) *ask* if the user would like to submit those results to be displayed in public.
> 
> Yes and yes. a) is not trivial with the current implementation (there is no real notion of "session" right now) but that's a good idea. It matches Philippe's request as well. And when we have that, we could easily add an export button to address Bryan's comment.
> 
> b) needs a), of course, and should at least be done before starting a "run" in the meantime. That's what I tried to do with the latest update. I agree it's not perfect.

No, there's no notion of a session in the harness, and I really don't want to add one on the server side, I'd be OK with a mode that stores results in browser local storage and submits them as a batch, but I don't want that to be the default mode (or required as it's designed to work with all sorts of clients and many will not support local storage).

The results display code does have a lot of filtering capabilities (most of which are not exposed in UI at the moment), i.e.: it can filter by browser, engine, platform, versions of all three and entered date (currently only results before a date, but adding an 'after' date would be trivial). Also all submitted results are tracked by user (IP for now, but there's a login system that's not exposed as well). It would be easy to display results from a given user within a given time window, so you could get a summary of your submitted results without adding session support (perhaps just set a cookie when starting runs to get the session start time?). Of course the results would already have been be submitted.

One item on my todo list (once the login system is exposed), is to allow a user to review their own results and retract them (or possibly edit them). Combined with start/end time filtering I think that should cover your needs here.

Peter

Received on Wednesday, 10 August 2011 18:31:53 UTC