Re: what to do after 10 evaluations (was: Discussions for testrun websites)

Hi Shadi, all,

especially if the time available is short, I urge EVAL TF to invite the public to test-use WCAG-EM  in parallel to our more controlled evaluation ASAP. After all, this is a document that claims to provide a general, public, and understandable instruction of how to evaluate web sites against WCAG. The view of outsiders will be especially valuable here as they may run into problems or misunderstandings that we as a task force will be unable to notice due to our familiarity with the WCAG-EM text.

Shadi, I don't understand what you mean with "then we can see if we still need more" (evaluation beyond the first 10 test runs). How would we ever find out if we need more? Any experience, misunderstanding, hint by accessibility practitioners will be highly valuable. It will also be different in nature from previous external comments as some problems only become apparent when you start *applying* an instruction to guide an actual activity. 

Any outside practical test-run feedback that is made available to WCAG-EM is an important input and will help this or another future task force improve the methodology. I simply fail to see why openly inviting the use of WCAG EM on self-selected websites (maybe using another instance of the same questionnaire) is in conflict with our simultaneous more controlled test run of the selected site. The type of input we will/may get is different but complimentary, and in my eyes at least as valuable.

So my suggestion is:

1 Prepare a short text announcing a public evaluation of WCAG EM, 
  publish that on the WAI site and make it widely known via the 
  usual channels (WCAG list, Webaim list, Twitter, etc). (The text can 
  also mention the more controlled evaluation going on at the same 
  time.)

2 Explain that the aim of the open test run is to capture a wide 
  range of different sites to evaluate the applicability of WCAG-EM
  to these types of sites (incl. web apps)

3 Explain that people can choose any site they like (for example, 
  their own site, sites they are going to evaluate anyway, site they 
  think are interesting, technically challenging, particularly good 
  or bad, etc) BUT in the knowledge that the results of their test run 
  won't be published and are merely intended to reveal problems 
  with / or options for improvement of WCAG EM, to be used by EVAL TF

4 Explain that there is no obligation to finish evaluations, feedback
  on incomplete evaluations by experts who don't have the time to go
  through the entire set of SCs can be very valuable 

5 Explain that the feedback will be evaluated and generalised results 
  be published without specific reference to the individuals and to sites 
  that were reviewed. How much work we will put into that is up to us.

I actually don't think we will get a deluge of feedback, but if we get more than the handful of evaluations that will be carried out by us 'regulars', this feedback will be highly important. It will tell us to what extent WCAG EM is actually of practical value for evaluators out there.

If you really fear that we will get a deluge, we could simply have a pre-registration step for people wanting to use the questionnaire, explain that slots are limited and first-come-first served, and draw a line after, say, 50 registrations. I doubt it will be so many anyway.

Best,
Detlev


On 10 Mar 2014, at 00:10, Shadi Abou-Zahra wrote:

> Hi Eric, all,
> 
> On 7.3.2014 23:46, Velleman, Eric wrote:
>> Discussion 4:
>> We do not require people to do full evaluations anymore after we receive 10 full evaluations. We could pick 10 success criteria that need to be evaluated. This saves a lot of time and makes it easier to find more people to do an evaluation.
>> 
>> What is in it for us?
>> More people may be willing to do an evaluation and it still helps us because we are testing WCAG-EM and not if people know how to test WCAG AA.
> 
> Let's first try to get 10 full evaluations in the short time that we have available, then we can see if we still need more. Depending on what kind of information we still need, we can consider changing the test-run accordingly. For now, let's try to get the test-run going.
> 
> Best,
>  Shadi
> 
> -- 
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Activity Lead, W3C/WAI International Program Office
> Evaluation and Repair Tools Working Group (ERT WG)
> Research and Development Working Group (RDWG)

-- 
Detlev Fischer
testkreis - das Accessibility-Team von feld.wald.wiese
c/o feld.wald.wiese
Thedestraße 2
22767 Hamburg

Tel   +49 (0)40 439 10 68-3
Mobil +49 (0)1577 170 73 84
Fax   +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites

Received on Monday, 10 March 2014 08:31:51 UTC