- From: Shadi Abou-Zahra <shadi@w3.org>
- Date: Mon, 04 Apr 2011 17:10:24 +0200
- To: public-test-infra@w3.org
Ref: <http://www.w3.org/2011/04/04-testing-minutes.html>
Best,
Shadi
Testing
04 Apr 2011
See also: [2]IRC log
[2] http://www.w3.org/2011/04/04-testing-irc
Attendees
Present
Philippe, Shadi, Francois, Michael, Mike
Regrets
Chair
Philippe
Scribe
Shadi
Contents
* [3]Topics
1. [4]Closing on the requirements
2. [5]Testing Interest Group charter (Plh and Mike)
* [6]Summary of Action Items
_________________________________________________________
Closing on the requirements
<plh> [7]http://www.w3.org/wiki/TestInfra/goals
[7] http://www.w3.org/wiki/TestInfra/goals
plh: separated a little more but did not add examples yet
... SAZ said that most accessibility tests may be manual
[8]http://www.w3.org/wiki/Testing/accessibility
[8] http://www.w3.org/wiki/Testing/accessibility
<francois> shadi: I added a page on accessibility testing where I
tried to expand a little more on the different types of
accessibility testing. Some tests can be automatically automated.
<francois> ... In particular, some ARIA test cases can be nicely
automated, I think.
plh: semi-automatic seem to be a mix of self-describing and
automatic
... may be similar to geo-location testing
... they would also require human intervention
... interesting case about how people would write their tests
... for some WAI tests the human provides the results whereas with
the geo-location the result is provided automatically based on what
a human does
mc: had thoughts but did not add them to the wiki yet
<MichaelC_> Test cases separate from test files
<MichaelC_> Test cases supported by 0 or more test files
<MichaelC_> A given test file can be used by 0 or more test cases
<MichaelC_> Test cases and test files can be shared among WGs
(implication of above requirements is that WGs could share test
files but have different test cases)
<MichaelC_> Metadata not stored in test files
<MichaelC_> Provision for passing and failing test cases
<MichaelC_> Support for automated execution of test cases
<MichaelC_> Support for manual execution of test cases
<MichaelC_> Easy way to store test results
<MichaelC_> (possibly future) way to store and compare test results
from different testers, and declare an "authoratative" result
<MichaelC_> Way to associate test cases with specific spec
requirement
<MichaelC_> Way to perform tests against multiple user agents
(defined broadly as UA on a platform, potentially with a given AT,
plugin, or other support tool running)
<MichaelC_> (possibly future) separate analysis against different
UA, different OS, different AT, different AAPI, etc.
<MichaelC_> Different test cases allowed for different UAs
<MichaelC_> Way for not-to-technical users to add tests
<MichaelC_> Way to add large amounts of test cases and/or test files
at once, e.g., because auto-generated from spec
<MichaelC_> Way to isolate each spec feature tested in order to
associate test cases with them
<MichaelC_> Metadata requirements: see TSDTF format
<MichaelC_> (possibly future) way for members of public to submit
tests for consideration (would need a vetting process)
<MichaelC_> Way to associate test cases with 1 or more spec features
(preference for unit tests associated with one feature, but
aggregated tests may be needed)
<MichaelC_> Types of tests: parsing, DOM (or other memory model)
effect, presentation, API exposure, reaction
saz: the break-down of semi-automated testing is useful to WAI too,
for instance for user input for script testing
... might be more relevant for test authoring but need to keep that
in mind
plh: mc, didn't understand the separating test file from test case
mc: let's say testing for the display of a particular attribute in a
browser vs testing how it is communicated to the API
... two different tests on the same test file
plh: should make the test automatic, can't separate the automation
mc: by separating test files from test cases, just an attribute in
the metadata
plh: adds a lot of complexity
mc: if just looking at one or two groups' needs for now, then may do
differently
... but on the long term, thought we need to support a W3C-wide
framework
[9]http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1.
0
[9]
http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1.0
plh: how will the test logic look like?
mc: by adding the logic into the test files you are making each a
small evaluation tool
... think it may be easier to create a single evaluation engine to
process script logic
plh: just need to write the assertions for testharness.js and then
you are done
saz: how did we end up with testharness.js?
plh: needed a simple approach to carry out a test
... DOM WG and others had something like this years ago
... problem is that other libraries are pretty heavy
<plh> test(function() {assert_true(true)}, "assert_true with true")
saz: are we starting with requirements then trying to find tools
that match these needs or the other way around?
plh: difficult enough to get people to write tests
saz: isn't testharness.js a type of framework in itself, and don't
people need to write logic anyway?
mc: also hard to port to other frameworks in the future
plh: not really
mc: haven't looked at testharness.js specifically but from
experience test metadata does interfere with test cases
... so at least these need to be separated
plh: have examples from web performance group and can tell you that
separating the logic makes it extremely difficult
... on the other hand, sometimes logic within the test case can get
in the way
... like checking the DOM where writing logic would actually change
the DOM
... need to support different methods, do not want to constrain
people
<Zakim> shadi, you wanted to ask about different methods
fd: could have different ways of authoring tests
... could provide a tool that will convert declarative tests into
testharness.js format
... not easy to do the other way around
... testharness.js would be the smallest common
mc: yes, it is possible but not sure what we are supposed to be
doing
fd: even if you do it on your own, it would become part of the
overall tool
<MichaelC> [10]ARIA Draft test plan
[10] http://www.w3.org/WAI/PF/wiki/ARIA_Test_Plan
mc: some ARIA tests may fall out of scope of testharness.js
fd: this would fall back into the scope of requirements
mc: would need to write some tests and bring back the requirements
plh: or at least one test
[11]http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1
.0
[11]
http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1.0
<MichaelC> side note, the work done by OpenAjax on ARIA testing is
something we expect to incorporate into the ARIA test plan, this
work is done by PFWG members anyways
saz: so we have the requirement to support both the declarative and
procedural approach, and a converter from declarative to procedural
plh: agree with requirement but have to be careful of scope
saz: was thinking of unicorn or some other framework
plh: unicorn is more to merge reports
<MichaelC> [12]http://www.w3.org/WAI/GL/WCAG20-TECHS/PDF1#PDF1-tests
[12] http://www.w3.org/WAI/GL/WCAG20-TECHS/PDF1#PDF1-tests
mc: example linked above, need to support many ways of carring out a
test
plh: this is what the requirements page needs to cover
... also need to specify what our requirements for the declarative
approach would be
... can quickly get out of hand
saz: suggest we need to refine the requirements a little bit more,
especially to add the differentiation between declarative and
procedural approach
<MikeSmith> many of the requirements we have already added on the
wiki page do not necessarily have any level of consensus either
mc: was not sure if should put requirements directly in the wiki or
if we need more discussion
Testing Interest Group charter (Plh and Mike)
plh: need to put them in now
ms: have a draft for the IG, very sketchy at this point
<MikeSmith> [13]http://www.w3.org/2011/05/testing-ig-charter.html
[13] http://www.w3.org/2011/05/testing-ig-charter.html
ms: appreciate comments and feedback
... need to have well-bounded group
saz: concern about the limited scope, more limited than the scope of
the Vision TF intended
plh: WAI-ARIA is missing. But I wouldn't want to see XQuery listed
ms: often have the issue of scope-creep so need to keep the scope
bounded
<francois> shadi: think mentioning types of tests might be more
useful than listing relevant technologies.
fd: think good idea to have an IG, think the list of specifications
is good for a start
<MikeSmith> I personally have never written up a charter like this
without listing specific technologies that the group's work is
intended to be related to; I think in terms of the technologies
first
saz: agree with an IG as well, but currently the scope would exclude
WCAG
<MikeSmith> we don't typically test server-side behavior; we test
for what responses/information the server sends back to the
client/browser
plh: it would be exponentially harder to expand the scope
saz: should look back at the requirements and see what overlap we
have
... but need to address WCAG and WAI-ARIA
<MikeSmith> hmm, the Web Notifications API is another case where the
behavior cannot be tested within a browser
<MikeSmith> because the behavior is that the browser causes some
platform-level notification to be generated
<MikeSmith> generated outside the browser
<plh> good point
<MikeSmith> the only thing we can test within the browser is whether
it actually fires the "show" event
<MikeSmith> oh, and whether it actually creates a Notification
object
Summary of Action Items
[End of minutes]
--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/ |
WAI International Program Office Activity Lead |
W3C Evaluation & Repair Tools Working Group Chair |
Received on Monday, 4 April 2011 15:11:04 UTC