W3C home > Mailing lists > Public > public-html-testsuite@w3.org > February 2013

Re: Marking non-automated tests

From: Robin Berjon <robin@w3.org>
Date: Thu, 07 Feb 2013 12:01:54 +0100
Message-ID: <511389A2.6020603@w3.org>
To: Wolfram Kriesing <wk@uxebu.com>
CC: "public-html-testsuite@w3.org" <public-html-testsuite@w3.org>, Tobie Langel <tobie@w3.org>
Hi Wolfram!

It's good to see you here :)

On 07/02/2013 09:58 , Wolfram Kriesing wrote:
> Maybe I introduce myself a bit. I am Wolfram Kriesing, from uxebu,
> a small JavaScript company. I had been working on an alike project
> a web test suite, that was open sourced by vodafone
> https://github.com/vf/web-testsuite

I wonder: could some of this be imported into our test suite?

> I am curious if there are any discussions or plans to
> 1) host the test for execution and (the imho important thing)

Yes. We have w3c-test.org for that, and as it happens the tests in 
html-testsuite are mirrored to http://w3c-test.org/html-testsuite/.

This isn't enough though. As it is, it's just a web server. We want to 
make it possible to automatically run all the tests that are automatic 
(duh) or a configurable subset thereof easily.

> 2) to collect all the data
> What do i mean? maybe it becomes obvious when running one of the
> simple tests we used to build in that old test suite
> http://static.uxebu.com/~cain/web-testsuite/src/tests-w3c-mediaquery-orientation.html
> the test ist manual and the results are sent back to a server, so
> we can generate stats about pass/fail on certain browsers
> and devices ... unfortunately we never generated any stats yet :(

That's also in the cards. In fact, we have an existing system:

     http://w3c-test.org/framework/app/suite

It knows about a bunch of test suites, has a runner, keeps the results, 
makes it possible to submit results for manual tests, etc.

We're not happy with it though, it has a number of problems. So over the 
coming months we're going to be overhauling all of this.

Tobie (copied) just joined the team in order to work on that (amongst 
other things).

> If this is a topic I would be interested in helping on it, I see a lot
> of potential and things to do there :)

Wonderful! Please coordinate with Tobie (he's been very busy switching 
from his previous role, and then sick, but I hope he'll be around to 
answer your questions more generally very shortly).

> Can I read up on the current plans
> regarding this somewhere?

I'm repeating myself here, but Tobie's working on that (and I'll be 
helping).

There was a testing meeting last week at which this was discussed 
(mostly from the point of view of planning, and getting resources). I 
think that PLH is still editing the outcome, but it should be posted 
today or tomorrow if it hasn't been already.

PS: sorry for lagging in previous replies to you on this, as you can see 
things are moving a fair bit in this area :)

-- 
Robin Berjon - http://berjon.com/ - @robinberjon
Received on Thursday, 7 February 2013 11:02:02 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 7 February 2013 11:02:03 GMT