- From: Francois Daoust <fd@w3.org>
- Date: Mon, 02 May 2011 16:53:32 +0200
- To: public-test-infra@w3.org
Hi,
Minutes of today's call are available at:
http://www.w3.org/2011/05/02-testing-minutes.html
... and copied as raw text below.
Thanks,
Francois.
-----
Testing team call
02 May 2011
Attendees
Present
plh, Francois, MikeSmith, Judy, Shadi, MichaelCooper
Regrets
Chair
plh
Scribe
francois
Contents
* [2]Topics
1. [3]requirements
2. [4]IG charter
* [5]Summary of Action Items
_________________________________________________________
requirements
<plh_> [6]http://www.w3.org/wiki/TestInfra/goals
[6] http://www.w3.org/wiki/TestInfra/goals
plh: Page for requirements. We need to wrap this up. We're already
late and need to move forward.
<MichaelC> [7]Michael take on requirements
[7] http://www.w3.org/wiki/Testing/Requirements/Cooper
michael: too many things going on at the moment. I've added my
comments in a separate page.
... in my email, I explain how to separate between use cases,
general requirements, and test files. From most of the requirements
I tried to detail things a bit. I know that I haven't captured
everything from the original page.
... There's still room for some merging, I haven't gone into
technical details.
plh: I'm skimming through it right now. It looks reasonable.
michael: I hope so. There's one requirement that there exist a
layered approach between test files and test cases.
... One other core requirement is that you can have test files that
are shared among working groups.
plh: already possible, yes.
michael: Too me, I thought this organization is better to grasp.
plh: things look good. We are working resources across working
groups already. I just take those for granted.
michael: I don't think we should take things for granted when
developing a product, we should be explicit.
judy: Even if the differences are not fully laid out here, we feel
it's important to talk about that together.
plh: happy to do that, but we need to spell out the differences.
shadi: very briefly, in the requirements that we have so far, the
question is how can we turn things that appear in the wiki into
something that we can put in the development of a product?
... I think we need to converge and try to find consensus on
specific requirements that we all share
judy: michael, could you lay out some of the differences?
michael: unclear to me what we have and don't have so far. For
instance, testharness.js is good but I don't think that we're going
to be able to reuse it for WAI-ARIA.
... There are test cases that cannot be expressed like that. I need
separation between metadata and test files.
... We may be needing manual tests here.
plh: That's what we've been saying in the past two months.
... We have the exact same requirements in the HTML working group.
Manual test cases will be needed. Metadata cannot be put in all test
files. I don't hear any difference.
michael: then it goes back to the requirements page that is
confusing.
<judy> [8]http://www.w3.org/wiki/TestInfra/goals#Self_describing
[8] http://www.w3.org/wiki/TestInfra/goals#Self_describing
plh: how is "self describing" test confusing?
shadi: where is the JavaScript one?
plh: one of our goals is to make tests as automated as possible. We
know that it's not always possible. We'll have thousands of tests in
HTML. We cannot afford to run these tests manually.
... There are examples there about what we mean by that.
... The framework that will be in charge of running the test cases
and collecting reports will not preclude any specific type of tests.
shadi: I'm still confused. Maybe it's just because it's named
"testharness.js"
... I see possibility to ask users for permissions. So this is not
fully automated?
plh: right, the result can be reported automatically, but the test
may involve manual user intervention.
shadi: so we might want to separate them
plh: maybe it's more that we should call them "automatic result
reporting"
... whether the script can run the test automatically or not is
independent on how it reports the result back to the framework.
shadi: another question. So we have this testharness.js which is
pretty far already. It covers certain parts of testing that we'll
want to reuse. What about the other types of testing?
... Are we going to develop another test harness for that?
plh: no, that will be part of the test framework, the thing that is
on top of the rest.
... The framework will collect the result automatically if it's an
automated test, perform the comparison if it's a ref test, or ask a
human to report on the result of the test.
shadi: ok, thanks for the clarification, that's what I would call
"harness", but not a big deal as long as we agree on definitions.
plh: whatever we're going to create in the end, I want to be as
flexible as possible. The state of the art is going to change in the
next two years.
... A new version of testharness.js is in the works.
... That's not a big deal, the important stuff is that they can be
run in the same testing framework.
... Human won't change, so manual tests are easier. The test could
be executed outside of the browser environments.
... The only requirement for manuel tests is running a browser
environment to report on the result.
[Michael back to the call. plh going through same explanation]
judy: are there any reasons why the test harness being developed
would not be a good fit for WAI-ARIA?
plh: The general harness for me is the framework.
... Let me share an example of such a framework.
<plh_> [9]http://w3c-test.org/html/tests/harness/harness.htm
[9] http://w3c-test.org/html/tests/harness/harness.htm
plh: That's not the best example that we have. Two parts when you
load this page. The top part is the framework running. The bottom
part is the test running.
... Looking at this, you should be able to tell whether the test has
passed or failed easily.
... In case of automated tests, the framework would just collect the
result and move on to next test.
... This harness has limitations: it does not know how to recognize
automated and manual tests.
... Aesthetically, it's not the best one either.
... In some examples, the test itself cannot tell you whether it
passed or failed.
michael: today is the first time that this is not the harness that
we'll be developing.
judy: some kind of miscommunication. Given that plh is saying that
i've heard that this is the kind of things that we'll be doing, does
that sound like going in the right direction for WAI-ARIA if this
thing is cleaned up?
shadi: Michael's requirements seem refinement of the test framework
plh: yes and no.
<judy> [judy suggests adding a glossary, for fear of additional work
time lost in misunderstandings]
judy: it may be worthwhile to design a glossary not to lose
additional time. It should be quick.
plh: Right. I tought I had settled down the terminology two weeks
ago, but I don't seem to have been very successful at that.
... I probably need to create a diagram.
judy: Before we get to the top of the hour, can we focus on merging
requirements?
... A quick assess on whether that's doable?
plh: It doesn't seem to be too complex to do the merging.
... I'm happy to take an action to try to put our requirements page
into Michael's page to help clarify things.
IG charter
<plh_> [10]http://www.w3.org/2011/05/testing-ig-charter.html
[10] http://www.w3.org/2011/05/testing-ig-charter.html
plh: we've been talking about the draft IG charter for about month.
... Goal is to review it internally before call for review.
shadi: a couple of comments. Thanks for taking previous comments on
the scope section.
... The paragraph before last paragraph in the scope section kind of
narrows the scope and links to the Wiki which may not be appropriate
in a charter.
plh: oh, we should be sending an advance notice to the AC as well.
Summary of Action Items
[End of minutes]
Received on Monday, 2 May 2011 14:53:59 UTC