- From: Francois Daoust <fd@w3.org>
- Date: Tue, 22 Mar 2011 14:22:50 +0100
- To: public-test-infra@w3.org
Minutes of yesterday's call:
http://w3.org/2011/03/21-testing-minutes.html
... copied as raw text below.
Discussion around:
- Shadi's updates to requirements page:
http://www.w3.org/wiki/Testing/Requirements
- Review open-source Web test suite:
http://webtestsuite.org/
Francois.
-----
21 Mar 2011
See also: [2]IRC log
[2] http://www.w3.org/2011/03/21-testing-irc
Attendees
Present
Plh, Shadi, francois
Regrets
Chair
plh
Scribe
plh, francois
Contents
* [3]Topics
1. [4]List of requirements
2. [5]Potential contributor
* [6]Summary of Action Items
_________________________________________________________
List of requirements
Francois: still looking at the Mozilla test framework. they are many
ways that are used to test firefox
... some are testing the chrome
... some are testing the display engine
... hard to differentiate
... so, still looking at my way in
... most of JS tests rely on having some extensions installed
shadi: the distinction between the chrome and the browser is
interesting
-> [7]https://developer.mozilla.org/en/Mozilla_automated_testing
Mozilla automated testing (for starters)
[7] https://developer.mozilla.org/en/Mozilla_automated_testing
Shadi: for test execution section, it's still a bit too vague
plh: I don't mind targeting a main context that is a Web browser
with HTML/CSS/JavaScript support.
shadi: there may different types of tests and different ways to
approach these tests, to run them.
plh: Yes, I agree. For screenshots, it may be JavaScript-based, or
using some external tool, or manual.
shadi: we should look at these classes of tests and precise them a
little bit, to see if we need different methods to run them.
plh: we've been playing with some of these for some time.
... e.g. comparing plain text output is not as easy to do
cross-platform. So that's why we separated from JavaScript testable
tests.
shadi: maybe we could link to examples from the Wiki in terms of
testing method
<scribe> ACTION: plh to add examples to the Wiki in terms of testing
methods [recorded in
[8]http://www.w3.org/2011/03/21-testing-minutes.html#action01]
shadi: back to the comment that main target has JavaScript enabled,
I think we may have limitations when we're testing JavaScript.
plh: we know already that there are tests that cannot be tested with
JavaScript.
... I just do not want to spend ages trying to see how to test a
browser that does not have JavaScript.
... Most CSS tests do not have JavaScript inside them. Many HTML
tests do not have JavaScript either. A good example is parsing
tests.
shadi: one of the issues is how assistive technologies play with the
DOM and we are willing, as part of ARIA, to test JavaScript support
for some aspects.
plh: yes, relying on JavaScript for some of the tests is a non
starter.
... Back at your inputs, I see you added media players. What do you
mean by that?
shadi: I was not only thinking about HTML5, more in terms of generic
captioning, e.g. video online.
plh: well, that's HTML5, in my view.
... I can think of a good example that is not HTML: SVG video.
... We may not have HTML around, but we still have JavaScript
around.
... as part of HTML testing, we're not going to test the properties
of plug-ins, inside the object element.
... That's not our job.
... If you guys define some generic constraints on video playback
because you want to be technology independent, that may be
different.
shadi: For instance, if a user tabs between elements in a Web page
that has some video running, no matter the method, it often breaks
the accessibility of the page.
plh: I would think that it is part of the HTML testing, as it is to
make sure that keyboard navigation behaves as expected, but the fact
that it's a video doesn't really matter.
... in the end, it will probably fall into human tests, i.e. I can't
think of a way to automate that kind of tests.
shadi: what about SMIL?
plh: not currently a priority.
... In terms of technologies, there is a subset of technologies that
I have in mind that range across 15 or so working groups.
... I could be wrong, but what you're suggesting around
accessibility APIs does not seem to be doable on an automated basis.
... manual tests we have in mind use a pass/fail button. It's easy.
It's our fallback in terms of testing method.
shadi: I did not see anything in the proposal that striked me as
"non-accessibility friendly", so more working groups may align to
this.
<plh-home> [9]http://w3c-test.org/resources/testharness.js
[9] http://w3c-test.org/resources/testharness.js
francois: wonder if we should put the set of dependencies that one
can use when one writes a test case, e.g. a Javascript test or a CSS
test.
plh: Interesting. For JavaScript, you can rely on the test harness I
just pasted.
... It could be used in a higher-level framework.
... I actually started to write tests using that harness on Friday,
because I realized there were a few troubles to be fixed before we
ask people to use it.
... For human tests, that's easy, you just explain the test to the
user.
... For screenshot tests, it's easy as well. Two files. The
comparison may be done automatically or manually.
... It's going to be part of the documentation we need to write for
working groups.
... We also need to give guarantees that we're not going to break
stuff like that.
... Back to your point, it means we need to make sure we have
documentation for it. I'm at least working on that aspect for
testharness.js
... there will be different approaches to writing tests in different
working groups.
... We're still learning, here.
Potential contributor
Francois: Had a call with DKA about potential contributions for
HTML5 tests
... they already open sourced their test suite
->
[10]http://lists.w3.org/Archives/Team/ercim-mobiweb/2011Mar/0054.htm
l fd email on webtestsuite
[10] http://lists.w3.org/Archives/Team/ercim-mobiweb/2011Mar/0054.html
<plh-home> [11]http://webtestsuite.org/
[11] http://webtestsuite.org/
Francois: it's released until an Apache license
... their initial goal is to test WAC
... about 900 test cases. Some of them test performance imposed by
WAC specs
... in the end, there are 200 tests for us
... HTML5, CSS, APIs, widgets, ...
... the tests are defined in JS
... they know that just open sourcing isn't enough
plh: would be good to have a documentation already. I guess the best
way to move forward is to tell them to join the task force, point
them to the test harness.
... The bottom line is that we have this Mercurial server where
people should drop their test cases that need to undergo a review
process.
... It's not perfect.
... one requirement we have is to keep the tests separated.
francois: ok, I'll get back to them.
plh: one possibility is to have some code that generate test cases
that we can integrate.
francois: right, that's one possibility we mentioned
plh: we don't want to have tests depend on the higher-level testing
framework, because such test suites tend to die in the long run
<plh-home>
[12]http://w3c-test.org/webperf/tests/approved/test_navigation_attri
butes_exist.htm
[12] http://w3c-test.org/webperf/tests/approved/test_navigation_attributes_exist.htm
plh: here's an example of a JavaScript test.
... It contains several tests, and reports. It's a test. It's
isolated, does not depend on a higher level framework.
... We're using the test harness, but that's, to some extent,
orthogonal to the thing we're testing.
... The higher level test framework will collect all the results
from all these tests and report them in some nice view or report.
... One possible problem with Webtestsuite tests is that they are
dependent on the higher level harness, so cannot be taken away from
it.
<plh-home>
[13]http://w3c-test.org/html/tests/approved/video/video_000.htm
[13] http://w3c-test.org/html/tests/approved/video/video_000.htm
plh: another example. No JavaScript involved. You have to read the
test to understand whether the test passed or failed. It's up to the
framework to provide some means to the user to indicate whether the
test passed or failed.
... It's not in the test itself.
... The test is independent on the framework.
... So if we decide to rewrite the framework in 10 years, it's fine.
shadi: many accessibility tests will be more of the manual type.
... these examples are making it much more concrete, so thanks for
sharing.
... I think Michael and I should be looking at accessibility tests
and see if they fit under that categorization.
... I would also encourage your groups to write a few early tests
that you think should be automatic and submit them to the HTML
testing suite TF.
... Another example is a "ref test".
<plh-home>
[14]http://w3c-test.org/html/tests/submission/W3C/bidi-markup-export
/html5-html/
[14] http://w3c-test.org/html/tests/submission/W3C/bidi-markup-export/html5-html/
plh: The two files are supposed to render exactly the same way. If
we cannot do that automatically, we can ask a human to report.
Another way is to do screenshots.
... We're not entirely sure if it's doable, or not.
<gsnedders> Each (major) browser has their own API for doing so,
though I don't think IE has anything public, but otherwise do-able
with per-browser harnesses.
[small discussion on ref tests vs. DOM comparison tests]
Summary of Action Items
[NEW] ACTION: plh to add examples to the Wiki in terms of testing
methods [recorded in
[15]http://www.w3.org/2011/03/21-testing-minutes.html#action01]
[End of minutes]
_________________________________________________________
Minutes formatted by David Booth's [16]scribe.perl version 1.135
([17]CVS log)
$Date: 2011/03/22 13:09:41 $
[16] http://dev.w3.org/cvsweb/~checkout~/2002/scribe/scribedoc.htm
[17] http://dev.w3.org/cvsweb/2002/scribe/
Received on Tuesday, 22 March 2011 13:23:19 UTC