Re: Test Framework Task Force

Hi Jeanne, 

Thank you for the email. Please see my answers inline.

On Wednesday, March 20, 2013 at 10:55 PM, Jeanne Spellman wrote:
> Tobie,
> 
> I realize that your priority is automated testing of browsers, but the 
> test framework would also be helpful to other working groups who may not 
> need the sophistication and automation, but whose testing work would be 
> greatly facilitated by a testing framework tool.Whatever you build 
> into the test framework, please take into consideration some of the 
> needs for manual testing, or at least, not preclude them, so that they 
> could be added later.

Yes, manual testing should be possible using the framework, although that clearly is not the primary goal of the framework. 

I actually strongly disagree with your premise that automation is a requirement of certain groups only and that others don't need it. Automation is not a requirement of working groups, it is a requirement whenever we want to make sure tests are run. In practice, automated tests get run on a daily basis, manual ones, pretty much never. Now I'm aware that certain areas require manual testing, but a lot of tests can be automated and we should strive to do so everytime we can.

I am strongly committed to finding solutions to automate accessibility testing of user agents, and has been an important preoccupation of mine over the last couple of weeks. I've notably been exploring whether WebDriver would be an adequate solution to automate WAI-ARIA testing. I don't have a definitive answer, but it seems to be an interesting option.
> I have been eager to use the testing framework for my working groups, 
> but the current framework is not oriented toward the more manual testing 
> needs of many accessibility guidelines and the non-browser testing of 
> the Authoring Tool Accessibility Guidelines (ATAG).

As discussed with Michael Cooper (cc'ed), this is out of the scope of the current effort which focuses strictly on user agent testing, not on content created by authors or authoring tools.

These two problem domain are so different that I don't see any value in trying to shoehorn both requirements into the same framework. Parts of the infrastructure could be shared, however. But I'm unsure which at this point.
> Although the wiki page is marked as out of date, the requirements of the 
> WAI domain are still relevant.
> http://www.w3.org/wiki/Testing/accessibility
> Since that wiki page was created, I have experimented with the
> framework, and have the following requests to make the framework
> useful to AUWG and UAWG:
> 
> 1) the ability to test and report by product, not by browser (e.g.
> conformance of Dreamweaver, Blue Griffin, MediaWiki, Wordpress, etc.)

As mentioned above, that's out of scope (those are authoring tools). 
> 2) the user interface and the reports meet WCAG 2.0 AA (the framework
> will be used by current working group members with disabilities).

Thanks for bringing that up. This of course makes a lot of sense. Admittedly, I have little experience in budgeting such requirements. Would you be able to assist me with this, or point me to someone which could help?
> 3) the framework allows import of specs that have letters in the section 
> numbering (ATAG has Part A and B, which results in section numbers like 
> A.2.3.1)

That shouldn't be an issue. 
> 4) at minimum, the framework test harness has an additional answer
> option of "not applicable" in addition to "skip" and the reporting
> appropriately collects the different responses. Allowing the test
> administrator to define the buttons for answering would be most helpful.

So this seems like something we would need to implement directly in testharness.js[1] or in a small wrapper around testharness.js.

I don't think this should be technically challenging to do.
> 5) the framework test harness records the evaluator (person) name and
> produce reports by evaluator

That shouldn't be an issue either, except perhaps in terms of the UI for it.
> 6) the framework test harness allows entry of text comments by the evaluator

It's not clear to me how this would work. Would this be a comment per test multiple comments per tests, or for the overall test suite? Would these comments be stored with the test results, or would they be displayed on every subsequent test run?

> Please feel free to contact me if you would like more context or have
> better ideas for how we could test.



My impression is that shoehorning requirements for manual testing and authoring tool testing into this framework is going to create a product that's disappointing for all use cases. And I don't think it would substantially lower costs either. I feel like I need to learn more about your requirements and use cases to see what the best solution would be,


As I mentioned to Michael during our conversation, I feel like there is a lot of value in finding ways to automate accessibility testing within the scope of this testing effort (testing user agents). Focusing on WAI-ARIA to start with is a great compromise and makes perfect sense given the other specs tackled in this effort.

Happy to setup a call to discuss this further.

Best,
 
--tobie
---
[1]: https://github.com/w3c/testharness.js

Received on Wednesday, 20 March 2013 23:48:13 UTC