Re: what to do after 10 evaluations

Hi Detlev,

I am not at all concerned that we will get too many evaluations but 
actually too few, especially since we are on a short time frame. I think 
we need to adjust to get the maximum feedback from a minimum amount of 
test runs in a minimal amount of time.

That said, I can live with the suggestions outlined below. The down side 
is that we would then not have directly comparable evaluation results. 
But if we are primarily looking for feedback on the clarity and 
practicality of WCAG-EM, then we can use different websites.

This would also help us avoid any potential false expectations or 
misunderstandings that "W3C will evaluate your website" or such.

Best,
   Shadi


On 10.3.2014 09:47, Detlev Fischer wrote:
> Hi Shadi, all,
>
> especially if the time available is short, I urge EVAL TF to invite the public to test-use WCAG-EM  in parallel to our more controlled evaluation ASAP. After all, this is a document that claims to provide a general, public, and understandable instruction of how to evaluate web sites against WCAG. The view of outsiders will be especially valuable here as they may run into problems or misunderstandings that we as a task force will be unable to notice due to our familiarity with the WCAG-EM text.
>
> Shadi, I don't understand what you mean with "then we can see if we still need more" (evaluation beyond the first 10 test runs). How would we ever find out if we need more? Any experience, misunderstanding, hint by accessibility practitioners will be highly valuable. It will also be different in nature from previous external comments as some problems only become apparent when you start *applying* an instruction to guide an actual activity.
>
> Any outside practical test-run feedback that is made available to WCAG-EM is an important input and will help this or another future task force improve the methodology. I simply fail to see why openly inviting the use of WCAG EM on self-selected websites (maybe using another instance of the same questionnaire) is in conflict with our simultaneous more controlled test run of the selected site. The type of input we will/may get is different but complimentary, and in my eyes at least as valuable.
>
> So my suggestion is:
>
> 1 Prepare a short text announcing a public evaluation of WCAG EM,
>    publish that on the WAI site and make it widely known via the
>    usual channels (WCAG list, Webaim list, Twitter, etc). (The text can
>    also mention the more controlled evaluation going on at the same
>    time.)
>
> 2 Explain that the aim of the open test run is to capture a wide
>    range of different sites to evaluate the applicability of WCAG-EM
>    to these types of sites (incl. web apps)
>
> 3 Explain that people can choose any site they like (for example,
>    their own site, sites they are going to evaluate anyway, site they
>    think are interesting, technically challenging, particularly good
>    or bad, etc) BUT in the knowledge that the results of their test run
>    won't be published and are merely intended to reveal problems
>    with / or options for improvement of WCAG EM, to be used by EVAL TF
>
> 4 Explain that there is no obligation to finish evaluations, feedback
>    on incomplete evaluations by experts who don't have the time to go
>    through the entire set of SCs can be very valuable
>
> 5 Explain that the feedback will be evaluated and generalised results
>    be published without specific reference to the individuals and to sites
>    that were reviewed. How much work we will put into that is up to us.
>
> I actually don't think we will get a deluge of feedback, but if we get more than the handful of evaluations that will be carried out by us 'regulars', this feedback will be highly important. It will tell us to what extent WCAG EM is actually of practical value for evaluators out there.
>
> If you really fear that we will get a deluge, we could simply have a pre-registration step for people wanting to use the questionnaire, explain that slots are limited and first-come-first served, and draw a line after, say, 50 registrations. I doubt it will be so many anyway.
>
> Best,
> Detlev
>
>
> On 10 Mar 2014, at 00:10, Shadi Abou-Zahra wrote:
>
>> Hi Eric, all,
>>
>> On 7.3.2014 23:46, Velleman, Eric wrote:
>>> Discussion 4:
>>> We do not require people to do full evaluations anymore after we receive 10 full evaluations. We could pick 10 success criteria that need to be evaluated. This saves a lot of time and makes it easier to find more people to do an evaluation.
>>>
>>> What is in it for us?
>>> More people may be willing to do an evaluation and it still helps us because we are testing WCAG-EM and not if people know how to test WCAG AA.
>>
>> Let's first try to get 10 full evaluations in the short time that we have available, then we can see if we still need more. Depending on what kind of information we still need, we can consider changing the test-run accordingly. For now, let's try to get the test-run going.
>>
>> Best,
>>   Shadi
>>
>> --
>> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
>> Activity Lead, W3C/WAI International Program Office
>> Evaluation and Repair Tools Working Group (ERT WG)
>> Research and Development Working Group (RDWG)
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Monday, 10 March 2014 08:58:42 UTC