Re: [eaccessnet] Call for Review: Website Accessibility Conformance Evaluation Methodology (WCAG-EM)

Dear Eval TF,

Please find some input on sampling and evaluating web applications from 
the i2Web Project. It has some interesting references too.

Regards,
   Shadi


On 26.10.2012 12:09, Christopher Power wrote:
> Hi Shadi,
>
> Thanks for the opportunity to contribute.
>
> This information comes from work done in the EU Project Inclusive Future Internet Web Services (i2Web, http://www.i2web.eu/index.html).  The project is about trying to provide new design adaptations for people with disabilities on the web, as well as building new tools for the stakeholders in web accessibility, including developers, accessibility experts and commissioners.  In the elicitation work we did with those stakeholders, we interviewed 14 accessibility experts about their practices from a variety of organisations.  The complete report is in our deliverable which can be found at the following link:
> http://www.i2web.eu/downloads/201201_I2Web_D32.pdf
>
> We noted in your call that you wanted some information about sampling methods.  I've looked at all of Section 3 of your Evaluation procedure, and overall, I think the information fits largely with the practice that we found.  The only major thing I would say that is different is that the activities that you list for sampling are often framed in terms of user journeys or user tasks throughout website by web accessibility evaluators.  The accessibility experts work with the commissioners and developers to identify what the key tasks are to users, and what the journey is that is taken through the website.  The following is an except from the deliverable (cleaned up a little bit with some referencing in preparation for a paper we are writing.
>
> I hope all is well, and if I can provide further information, please let me know.
>
> Cheers,
> Chris
>
> --
> There were very few accessibility experts who do evaluations on all web pages in any given web application or site. In some cases, this is due to the fact that there are thousands of pages within the web application/site being tested; however, more commonly it was because the commissioners of the web applications did not want all pages tested due to the cost of testing on such a large scale.
>
> As a result, most web accessibility experts test a sample of pages from a web application/website using a variety of different sampling strategies.
>
> First, all experts mentioned having discussions with the web commissioners or developers about what the most important pages are in the web application/site to be tested.  In this application-centric sampling strategy, the definition of “important pages” changed depending on the expert and the client with whom he/she is working.  For example, an important page might be one that has critical information about the organization.  Alternatively, it could be one that has potentially “tricky interactive components” such complex forms, Flash animations.
>
> 12 of the experts talked about a template-centric sampling strategy, where they identify a collection of pages that have a variety of different templates that are shared throughout a web application/site.  This was often referred to as a “representative set” of pages by interviewees.  Examining pages that share a particular template and identifying those with complex interactions then adds to this representative set. This strategy is a variation on the ad hoc sampling strategy discussed in various sources (Velleman, Velasco, Snaprud and Burger, 2006; Brajnik, Mulas and Pitton, 2007; W3C/WAI, 2011).
>
> One common strategy mentioned by 9 of the experts for selecting web pages within a web application/site is that of using user goals or, as some experts referred to it, “user journeys” in a website.  This user-centric strategy involves working with the development team or those who commissioned the website to identify key user goals and tasks in the application/site.  These goals are then used to identify path that users will take through a website to achieve the goal.  For example, in a banking website a key feature would be for users being able to find out their current balance.  With such a goal in mind, an expert records all pages that are along the expected paths of interaction from the home page to that end goal.  A number of these types of goals are used to generate a larger sample of pages tested in a web accessibility assessment.  When asked why they use this strategy, the interviewees who used it responded unanimously that it was because such strategies make it easier to 
communicate with the web developers and commissioners about where problems are in a web site. One evaluator stated:
>
>   “[We] always try to have testing be [related to] the use of a specific task to see if there are problems along the critical path so that you can say to a commissioner ‘A user can’t buy something on your page’ rather than ‘there is a missing alt text’.”
>
>   No interviewees discussed using random sampling strategies such as those discussed in Henzinger et al. (2000) and Brajnik et al. (2007). "
>
> References
> Brajnik, G., Mulas, A. and Pitton, C. (2007). Effects of sampling methods on web accessibility evaluations. Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, Tempe, Arizona, USA.
>
> Henzinger, M. R., Heydon, A., Mitzenmacher, M. and Najork, M. (2000). On near-uniform url sampling. Computer Networks and ISDN Systems, 33(1-6), 295-308. doi: 10.1016/s1389-1286(00)00055-4
>
> Velleman, E., Velasco, C. A., Snaprud, M. and Burger, D. (2006). Unified web evaluation methodology (UWEM 1.2). Retrieved April 25, 2011, from http://www.wabcluster.org/uwem1_2/
>
> W3C/WAI. (2011). Conformance evaluation of web sites for accessibility. Retrieved April 25, 2011, from http://www.w3.org/WAI/eval/conformance.html
>
> --------------------------------------------------------------------------------------------------
> NB: New Email Address - please use
> christopher.power@york.ac.uk
> --------------------------------------------------------------------------------------------------
> Christopher Power, PhD.
> Lecturer
> Human Computer Interaction Research Group
> CSE/241, Department of Computer Science
> University of York,  Deramore Lane, York YO10 5GH
> Phone: +44 (0)1904325673 Fax: +44 (0)1904 432767
> Email: christopher.power@york.ac.uk;  Skype: cdspower; Twitter: cdspower
> Web: http://www.cs.york.ac.uk/~cpower
> --------------------------------------------------------------------------------------------------
> UoY Email Disclaimer
>
>
>
>
>
>
>
>
>
> On 23 Oct 2012, at 08:10, Shadi Abou-Zahra wrote:
>
>> Hi Chris,
>>
>> We'd love to get your input on evaluating web applications.
>>
>> We have a face-to-face meeting on Monday and Tuesday next week. It would be great if you can get us your comments before the end of this week so that we can look at them before the meeting.
>>
>> Many thanks,
>>   Shadi
>>
>>
>> On 23.10.2012 08:49, Christopher Power wrote:
>>> Hi Shadi, Eric,
>>>
>>> I hope this email finds you well.  I'm in a project called I2Web, which is looking at accessibility of Web 2.0 applications in a variety of ways.  One thing that we have is a set of 14 interviews web web accessibility experts who are working in the field.  I notice that your call has a request for information about sampling methods - we actually have a bunch of information about how that is being done in the field.
>>>
>>> I realise that we have missed the deadline on this, but is it still possible to submit comments to you guys on this material?
>>>
>>> Cheers,
>>> Chris
>>> --------------------------------------------------------------------------------------------------
>>> NB: New Email Address - please use
>>> christopher.power@york.ac.uk
>>> --------------------------------------------------------------------------------------------------
>>> Christopher Power, PhD.
>>> Lecturer
>>> Human Computer Interaction Research Group
>>> CSE/241, Department of Computer Science
>>> University of York,  Deramore Lane, York YO10 5GH
>>> Phone: +44 (0)1904325673 Fax: +44 (0)1904 432767
>>> Email: christopher.power@york.ac.uk;  Skype: cdspower; Twitter: cdspower
>>> Web: http://www.cs.york.ac.uk/~cpower
>>> --------------------------------------------------------------------------------------------------
>>> UoY Email Disclaimer
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Begin forwarded message:
>>>
>>>> From: Shadi Abou-Zahra <shadi@w3.org>
>>>> Date: 21 September 2012 01:10:40 GMT+01:00
>>>> To: eaccessnet@fit.fraunhofer.de
>>>> Subject: [eaccessnet] Call for Review: Website Accessibility Conformance Evaluation Methodology (WCAG-EM)
>>>> Reply-To: eaccessnet@fit.fraunhofer.de
>>>>
>>>> Dear eAccess+ Partners,
>>>>
>>>> Please find below the formal announcement for an updated Working Draft of W3C/WAI Website Accessibility Conformance Evaluation Methodology (WCAG-EM), open for public comments until 20 October 2012.
>>>>
>>>> We welcome your comments on this work.
>>>>
>>>> Regards,
>>>>   Shadi
>>>>
>>>>
>>>> -------- Original Message --------
>>>> Subject: 	Call for Review: Website Accessibility Conformance Evaluation Methodology (WCAG-EM)
>>>> Date: 	Thu, 20 Sep 2012 10:54:53 -0500
>>>> From: 	Shawn Henry <shawn@w3.org>
>>>> To: 	WAI Interest Group <w3c-wai-ig@w3.org>
>>>> CC: 	Shadi Abou-Zahra <shadi@w3.org>, Eric Velleman <e.velleman@bartimeus.nl>
>>>>
>>>>
>>>>
>>>> Dear WAI Interest Group Participants,
>>>>
>>>> WAI invites you to comment on the updated Working Draft of Website Accessibility Conformance Evaluation Methodology (WCAG-EM) at:
>>>>     http://www.w3.org/TR/WCAG-EM/
>>>>
>>>> Overview:
>>>> WCAG-EM describes an approach for evaluating how websites -- including web applications and websites for mobile devices -- conform to Web Content Accessibility Guidelines (WCAG) 2.0. It covers different situations, including self-assessment and third-party evaluation. It is independent of particular evaluation tools, web browsers, and assistive technologies.
>>>>
>>>> Draft:
>>>> This draft has proposed content for all sections. Now is a good time to review it. We are particularly looking for feedback on the applicability of this methodology in practice and for input on refining and expanding the guidance provided. Specific questions are indicated with "Review Note" in the draft.
>>>>
>>>> Comments:
>>>> Please send comments on this draft document to the publicly archived mailing list:
>>>> 	public-wai-evaltf@w3.org
>>>> 	by *20 October 2012*
>>>>
>>>> Background:
>>>> WCAG-EM is developed by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF), a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG). The Eval TF is introduced at:<http://www.w3.org/WAI/ER/2011/eval/eval-tf>
>>>> It is a supporting resource for WCAG 2.0 and does not replace or supersede it in any way. For an overview of WCAG, see<http://www.w3.org/WAI/intro/wcag>
>>>> This work is developed with support of the EC-funded WAI-ACT Project (IST 287725) described at:<http://www.w3.org/WAI/ACT/>
>>>> It is part of W3C WAI activities on web accessibility evaluation and testing introduced at:<http://www.w3.org/WAI/ER/2011/eval/>
>>>>
>>>> URI:
>>>> The first URI above goes to the latest version of the document. The "dated" version of this draft is:<http://www.w3.org/TR/2012/WD-WCAG-EM-20120920/>
>>>> The difference between these URIs are explained in Referencing and Linking to WAI Guidelines and Technical Documents at:<http://www.w3.org/WAI/intro/linking.html>
>>>>
>>>> Please let us know if you have any questions. Thank you in advance for your comments.
>>>>
>>>> Feel free to circulate this message to other lists; please avoid cross-postings where possible.
>>>>
>>>> Regards,
>>>> ~Shawn Lawton Henry, WAI Outreach
>>>> Eric Velleman, Eval TF Facilitator
>>>> Shadi Abou-Zahra, W3C/WAI Staff Contact
>>>> --
>>>> eaccessnet mailing list
>>>> eaccessnet@fit.fraunhofer.de
>>>> https://mail.fit.fraunhofer.de/mailman/listinfo/eaccessnet
>>>
>>>
>>
>> --
>> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
>> Activity Lead, W3C/WAI International Program Office
>> Evaluation and Repair Tools Working Group (ERT WG)
>> Research and Development Working Group (RDWG)
>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Friday, 26 October 2012 15:52:14 UTC