Re: Tools and Test Data: WCAG-EM Report Tool feedback

Hi Phill,

Thank you for this valuable feedback! A few comments inline below:


On 27.5.2015 16:44, Phill Jenkins wrote:
> I see two major pieces of information missing from the methodology and
> reporting tool:
>
> 1. Scope (what is also called the "requirements step" by other
> professionals) is missing the Tools and audit techniques that will be
> used.  It is critical that this is agreed to up front and documented in
> the "plan" since not all tools and techniques results in the same
> findings!  Tools and techniques are divided into 3 types:
>          1. Semi-automated Checking tools such as Deque WorldSpace, SSB
> Bart AMP, IBM Mobile Accessibility Checker, Wave, HTML code validations,
> and others
>          2. Compatibility Testing using assistive Technology such as JAWS,
> NVDA, VoiceOver, TalkBack, etc.
>          3. Subject Matter Expert (SME) Inspection using software tools and
> simulators such as Firefox Accessibility Extension, IE Toolbar, source
> code inspections, etc.
>
> Note: that these three types are not the same as doing an end-user
> assessment using participants with disabilities.  User research and design
> evaluations are scoped very differently than accessibility verification
> test (AVT), accessibility assessments and audits.  User Expereince (UX)
> research is done with designs (e.g. wireframes) and/or prototypes, while
> AVT is done with pre or post production code.

What you call compatibility testing is meant to be recorded under 
"accessibility support baseline" in Step 1 of the tool. WCAG-EM 1.0 
encourages that the inspection tools and methods used are recorded, much 
like you describe, but does not require it. This information should be 
recorded under "additional evaluation requirements" also in Step 1. You 
can find more information about these terms in WCAG-EM:
  - http://www.w3.org/TR/WCAG-EM/


> 2. Test Data: The userIDs, passwords, and data needed to explore the web
> app or mobile app.  I'm assuming that "URLs section" includes the test
> server and test enviromments needed to conduct the evaluations; but a note
> should be added to make that explicit.  Test Data typically indlues data
> such as account numbers, zip codes, values that create certain screens,
> and data that causes error conditions.  All of these need to be gathered
> and documented.  Typicall documented in the Scope step and must be
> included before the Explore step can be completed.

You will find a note about that at the start of Step 2 of WCAG-EM, and 
also optional recording in Step 5 (under "evaluation specifics").


> Without this level of robustness, the evaluation methodology is not
> repeatable and the reporting tool's validity and value is suspect.

It seems that most of your concerns are addressed by WCAG-EM (though 
WCAG-EM is more flexible for different types of evaluation settings, and 
is also less prescriptive as a non-normative document). However, the 
Report Tool itself seems less clear, especially for people who're not 
fully aware of the content of WCAG-EM. This is my key take-away.

We are trying to get resources to continue our work on this tool, to 
address your and other comments that we received on this first version.

Thank you for taking the time to provide this valuable feedback!

Best,
   Shadi


> ____________________________________________
> Regards,
> Phill Jenkins,
> IBM Accessibility
> ----- Forwarded by Phill Jenkins/Austin/IBM on 05/27/2015 09:19 AM -----
>
> From:   Sharron Rush <srush@knowbility.org>
> To:     w3c WAI List <w3c-wai-ig@w3.org>
> Date:   05/27/2015 09:02 AM
> Subject:        CORRECTION: WCAG-EM Report Tool feedback welcome
>
>
>
> Hello again,
>
> I hope some of you have been looking at the Report Tool. I made an error
> in the suggested feedback pathway. and here is the corrected information.
> Rather than clutter the list, please send your comments through the
> channel noted in the footer of the Tool itself:
>
>   "Feedback: We welcome ideas, bug reports, and comments via GitHub or
> e-mail to wai-eo-editors@w3.org (a publicly archived list) or wai@w3.org
> (a WAI staff-only list)."
>
> Thanks, we hope to hear from you!
>
> Best,
> Sharron
>
>
>
> On Tue, May 26, 2015 at 12:02 PM, Sharron Rush <srush@knowbility.org>
> wrote:
> Greetings all,
>
> In March, WAI released the first version of the WCAG-EM Report Tool. [1]
> We got some useful feedback at AccessU earlier this month and are
> interested in more information from our community about how the tool is
> being used.
>
> Please send email here or to me off-list if you prefer. We are seeking
> input about how the tool is useful in your assessment work, what barriers
> you may have found to understanding or using it, and what would improve
> the tool for your situation.
>
> The plan is to gather feedback and use it for the next iteration of the
> Report Tool.  Thanks for any information you can provide.
>
> [1] http://www.w3.org/WAI/eval/report-tool/#/
>
> Best,
> Sharron
> ---
> Sharron Rush
> Co-chair, EOWG
>
>
>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Wednesday, 27 May 2015 15:14:35 UTC