W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > May 2012

Re: evaluating web applications (was Re: Canadian Treasury Board accessibility assessment methodology)

From: Peter Korn <peter.korn@oracle.com>
Date: Wed, 23 May 2012 11:12:24 -0700
Message-ID: <4FBD2888.509@oracle.com>
To: Shadi Abou-Zahra <shadi@w3.org>
CC: Eval TF <public-wai-evaltf@w3.org>
Shadi,

I can't speak to "usual practice" for web accessibility evaluators.  I 
think that software organizations generally who are implementing 
accessibility support simply test for it alongside general software 
testing.  E.g. on a regular basis as new internal builds are released to 
testing, QA will look at things like keyboard operability, theme 
support, color/contrast issues, and interoperability with assistive 
technologies.  Programmatic testing will be a part of this as much as 
possible.  QA practices to ensure code coverage generally would also 
apply to accessibility testing -> focus on those places where 
engineering has indicated new features were added or UI has changed; run 
some set of "sanity tests" to ensure that the most critical and basic 
functionality isn't broken (e.g. always try to read a known good sample 
web page each time you get a new build of Firefox), etc.

But this is different from a one-time review of an application; this is 
about the software development process and working to ensure 
accessibility is there - like any other feature and like overall 
functionality - as the product is being built.  I don't know how to 
adapt that process - which may involve hundreds of hours over a 
multi-year development cycle - to a one-short WCAG evaluation of a web 
application.


Regards,

Peter

On 5/23/2012 12:30 AM, Shadi Abou-Zahra wrote:
> Hi all,
>
> Is there a suggested approach/procedure for sampling functionality 
> within an application, as we have for web pages within a website? Is 
> this usual practice that web accessibility evaluators take?
>
> Best,
>   Shadi
>
>
> On 22.5.2012 23:24, Peter Korn wrote:
>> Shadi,
>>
>> I don't believe one can make an effective, useful, meaningful 
>> conformance claim
>> <http://www.w3.org/TR/WCAG/#conformance-claims>  about many classes 
>> of web
>> applications today. That class includes things like web mail, and 
>> many kinds of
>> portal applications (particularly where they only employ a single URI).
>>
>> I do believe it will be possible to evaluate web applications for 
>> accessibility
>> - similar to evaluating non-web applications for accessibility - but 
>> I expect we
>> will need to do something that is different from the binary
>> "perfection"/"imperfection" of the current conformance claim
>> <http://www.w3.org/TR/WCAG/#conformance-claims>  rubric. The Canadian 
>> Treasury
>> Board example takes a step along that path in shifting from one binary
>> "perfection"/"imperfection" statement to a two tiered, percentage 
>> collection of
>> 38 binary "perfection"/"imperfection" statements. But we need to go 
>> further than
>> that.
>>
>> I think the components of such a successful evaluation will need to:
>>
>>    * Recognize (as EvalTF is already doing) that only a 
>> sampling/subset of
>>      everything that a user can encounter can be effectively 
>> evaluated in a
>>      finite and reasonable amount of time
>>    * Provide greater granularity in the evaluation reporting - one 
>> that is
>>      designed to accommodate the reality of imperfect software while 
>> nonetheless
>>      providing useful information to those consuming the evaluation 
>> report such
>>      that they can make informed decisions based on it
>>    * Incorporate the concepts (as EvalTF is starting to do) of uses 
>> (or use
>>      cases) of the application so that the evaluation is meaningful 
>> in the
>>      context of how the web application will be used
>>
>>
>> I am eager to get further into these discussions in EvalTF, some of 
>> which may be
>> logical things to discuss as we review feedback from the public draft 
>> (including
>> some of the Oracle feedback... :-). And as I mentioned, we've already 
>> started
>> exploring some of this already.
>>
>>
>> Peter
>>
>>
>> On 5/22/2012 2:09 PM, Shadi Abou-Zahra wrote:
>>>  Hi Peter,
>>>
>>>  Does that mean that web applications cannot be evaluated?
>>>
>>>  Best,
>>>  Shadi
>>>
>>>
>>>  On 22.5.2012 20:40, Peter Korn wrote:
>>>>  Shadi,
>>>>
>>>>  As is clear from the Notes&  Examples under their definition of 
>>>> "Web page" at
>>>>  the bottom of the URL you circulated (below), it is clear they are 
>>>> looking to
>>>>  assess on a Pass/Fail basis the full complexity of web 
>>>> applications. As we've
>>>>  explored in recent EvalTF meetings, that is a very challenging 
>>>> thing to do,
>>>>  given how dynamic web applications can be (cf. their examples of a 
>>>> "Web mail
>>>>  program" and a "customizable portal site"). It is challenging in 
>>>> normal software
>>>>  testing to determine whether you have reached every possible code 
>>>> path&  every
>>>>  possible configuration of the structure behind a single URI, let 
>>>> alone answer
>>>>  Pass/Fail for each and every WCAG A/AA SC for those.
>>>>
>>>>
>>>>  Regards,
>>>>
>>>>  Peter
>>>>
>>>>  On 5/22/2012 6:10 AM, Shadi Abou-Zahra wrote:
>>>>>  Dear Group,
>>>>>
>>>>>  Ref:<http://www.tbs-sct.gc.ca/ws-nw/wa-aw/wa-aw-assess-methd-eng.asp> 
>>>>>
>>>>>
>>>>>  David MacDonald pointed out the accessibility assessment 
>>>>> methodology of the
>>>>>  Canadian Treasury Board, in particular the scoring they use.
>>>>>
>>>>>  Best,
>>>>>  Shadi
>>>>>
>>>>
>>>>  --
>>>>  Oracle<http://www.oracle.com>
>>>>  Peter Korn | Accessibility Principal
>>>>  Phone: +1 650 506 9522<tel:+1%20650%20506%209522>
>>>>  Oracle Corporate Architecture Group
>>>>  500 Oracle Parkway | Redwood City, CA 94065
>>>>  -------------------------------------------------------------------------------- 
>>>>
>>>>  Note: @sun.com e-mail addresses will shortly no longer function; 
>>>> be sure to use:
>>>>  peter.korn@oracle.com to reach me
>>>>  -------------------------------------------------------------------------------- 
>>>>
>>>>  Green Oracle<http://www.oracle.com/commitment>  Oracle is 
>>>> committed to
>>>>  developing practices and products that help protect the environment
>>>
>>
>> -- 
>> Oracle<http://www.oracle.com>
>> Peter Korn | Accessibility Principal
>> Phone: +1 650 506 9522<tel:+1%20650%20506%209522>
>> Oracle Corporate Architecture Group
>> 500 Oracle Parkway | Redwood City, CA 94065
>> -------------------------------------------------------------------------------- 
>>
>> Note: @sun.com e-mail addresses will shortly no longer function; be 
>> sure to use:
>> peter.korn@oracle.com to reach me
>> -------------------------------------------------------------------------------- 
>>
>> Green Oracle<http://www.oracle.com/commitment>  Oracle is committed to
>> developing practices and products that help protect the environment
>

-- 
Oracle <http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment
Received on Wednesday, 23 May 2012 18:13:04 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:14 GMT