RE: Input for survey about random sampling

Hi Shadi,

Thank you for the good comments and proposals for change. You are right, there are indeed a number of repetitions that should be solved first. Also I would like to try to minimize a bit more on the number of open questions as they tend to take a lot of time to answer. Will go through the questions this afternoon and try to come up with a new version. In the meantime, do not forget to enjoy your holiday.. 
Kindest regards,

Eric


________________________________________
Van: Shadi Abou-Zahra [shadi@w3.org]
Verzonden: vrijdag 12 oktober 2012 23:41
Aan: Velleman, Eric
CC: public-wai-evaltf@w3.org
Onderwerp: Re: Input for survey about random sampling

Hi Eric,

Thanks for the great work collecting these questions! I think this looks
good but not completely ready to go for several reasons.

Foremost I'm concerned that there are several repetitions throughout the
survey which may frustrate people and make it harder for them to
complete the survey. We should make sure the survey flows as smoothly as
possible to encourage people to contribute to it effectively.

I think that with a couple of minor edits you may be able to align the
questions and make the survey more compact and appealing. Please find
several comments for your consideration inline below:


On 12.10.2012 16:02, Velleman, Eric wrote:
> Dear all,
>
> Please find below a new version of the survey with changes made after our Telco. I also tried to work the comments received through the mailing list into the questions. Hope this would be ok. Think it would give us a good overview of activities. I added a number of yes/no questions to make it easier to calculate the results...
>
> Should we also add questions about the person answering the questions?
>
> Please send your comments to the list before monday evening.
>
>
>
> ** EvalTF Concept survey questions about sampling **
>
> Note: You can fill in the survey more than once using different choices in the first and second question. Please note that the Methodology is about full websites (as defined in the Public Working Draft). The Public Working Draft can be found at: http://www.w3.org/TR/WCAG-EM/
>
> #01. Please choose the evaluation activity for which you fill in this survey. If you recognize more than one of the activities, please consider filling out the form for each of them separately.
> Your evaluation activity is:
>    - (checkbox) In-house website evaluation for your own organization
>    - (checkbox) Website evaluation for external evaluation commissioner
>    - (checkbox) A re-evaluation of a website that you have evaluated less than 1 month ago
>    - (checkbox) Other - please specify below (text)

Is "1 month" important? Otherwise maybe just "Re-evaluation of a website
that you have recently evaluated"?


> #02. How would you describe the goal of your evaluation. For an explanation of the diferent goals, please read the section about goals of an evaluation: http://www.w3.org/TR/WCAG-EM/#step1b
>    - (checkbox) Basic evaluation
>    - (checkbox) Detailed evaluation
>    - (checkbox) In depth evaluation
>    - (checkbox) Other - please specify below (text)

Typo: "different goals" (one "f" missing).


> Please fill in the following questions depending on the evaluation activity you have chosen in question #01:

I think this could apply to question #2 as well, so no need for it.


> #03. For which purposes do you evaluate websites:
>    - (checkbox) Provide in-depth analyses to guide website owners
>    - (checkbox) Monitor the accessibility of a website over time
>    - (checkbox) Final check before releasing or purchasing a website
>    - (checkbox) Provide label or other certificate of conformance
>    - (checkbox) Do large-scale evaluation of many websites
>    - [checkbox] Pre-sales activities (evaluating websites in order to initiate a contract)
>    - [checkbox] Evaluate websites in early/middle development stages
>    - [checkbox] Deliver training about Web accessibility evaluation
>    - (checkbox) Other - please specify below (text)
>
> #04. Do you consult with the evaluation commissioner regarding the sample selection?
>    -(radio) Always
>    -(radio) Sometimes
>    -(radio) Never
>    -(radio) Depends upon whether the website is being evaluated with or without the knowledge of the evaluation commissioner

How about: "Depends on ... - please specify below" unless you really
only want to confine the question to this particular situation.

Note: I also think it should be "without the knowledge of the website
owner" rather than the "evaluation commissioner".

Please link the term "website owner" to the definition in WCAG-EM as
some readers may not understand what we mean exactly.


> #05. To what extent do the wishes of the evaluation commissioner influence the evaluation sample? 1= no influence (evaluator is independent in choice of evaluation sample), 5=total influence (evaluator is not independent in choice of of evaluation sample)
>    -(radio) 1
>    -(radio) 2
>    -(radio) 3
>    -(radio) 4
>    -(radio) 5

Maybe we should add a reminder that "evaluation commissioner" is not
necessarily the "website owner" and link these two terms to WCAG-EM.


> #06. Do you select sample pages for evaluation based on (note: this question is independent of the tools you use):
>    - (radio) Structured sampling only (Selection based on certain criteria)
>    - (radio) Random sampling only (Random selection)
>    - (radio) Both structured and random sampling
>
> #07. Do you use automated tools to help select the sample (crawlers etc.)?
>    - (radio) Yes
>    - (radio) No
>
> #08. If yes, please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

I suggest removing "if yes" to avoid limiting to automated tools only.
Maybe people use other ways (such as search engines) to select pages.


> #09. If you use automated sampling only, how do you check the tool? (text)

What do you mean by "check the tool"? Maybe "how do you ensure that the
suggested sample is indeed representative of the website"? This might
also be slightly redundant to questions #7 and #8.


> #10. If you use structured sampling, how do you select the individual web pages? (text)

Maybe refine that to "what criteria do you use to select the ..."?


> #11. If you use random sampling, how do you select the individual web pages? (text)
>
> #12. Does re-evaluation of a website influence the sample?
>    - (radio) Yes
>    - (radio) No
>
> #13. If yes, how does re-evaluation of a website influence the sample? (text)
>
> #14. Please specify if you sample pages differently or in different amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation? Note that the evaluations are in all cases full evaluations of a website. (text).
>
> #15. What is the amount of pages you use in your sample when evaluating a website?
>    - (radio) 1 - 5
>    - (radio) 6 - 15
>    - (radio) 16 - 30
>    - (radio) 31 - 50
>    - (radio) 51 – 100
>    - (radio) More than 100

Suggest changing "do you use in your sample" to "do you typically use in
your sample" in case it is not always a fixed number.

Might also be good to add "Other - please specify below", for example
for those who use percentages or other approaches.


> #17. You evaluate all the pages:
>    - (radio) Manually
>    - (radio)  Automatically
>    - (radio)  Automatically and manually
>    - (radio)  Automatically and a smaller sample manually
>
> #18. If the amount of pages in your evaluation sample is variable, what are –in most cases – the factor(s) that have the most influence on the size of the sample:
>    - [checkbox] Size of the website
>    - [checkbox] Number of technologies on the website
>    - [checkbox] Amount of money available for the evaluation
>    - [checkbox] Number of templates
>    - [checkbox] Other, please specify below (text)

Suggest adding "Complexity of website functionality" and changing
"Number of templates" to "Variety of web page types and styles", or to
add that latter question at least.


> #19. How does the size of the website influence your selection? (text)
>
> #20. How do you adjust your sampling approach for websites that are heavily template-driven? (text)

Questions #19 and #20 seem to overlap with question #18. What do we want
to get out of these questions and can we make them more specific?


> #21. If you use automated testing tool(s), does that change your sampling methodology?
>    - (radio) Yes
>    - (radio) No
>
> #22. If yes, please explain how the use of automated testing tools changes your sampling methodology (text)

Questions #21 and #22 repeat several other questions that we already
have earlier. Please align the questions and drop any redundant ones.


> #23. How do you adjust your sampling approach for client-side rich web applications? (text)
>
> #24. How do you sample mobile web applications? (We are not referring to native applications) (text)

How about combining these two into one question: "How do you address web
applications (including client-side rich web applications and mobile web
applications) in your sampling?"


> #25. Do you look out for widgets, snippets, and other web page components to avoid re-evaluating them every time they appear on a website?
>    - (radio) Yes
>    - (radio) No
>
> #26. If Yes, how do you include them and the pages that they are integrated in into the sample? (text)
>
> #27. How do you identify and sample the functionality (processes) provided on a website? (text)

I think this should come just before the question on web applications.


> #28. Do you select further sample pages based upon results from an initial selection (eg. to balance web page types or success criteria)?
>    - (radio) Yes
>    - (radio) No
>
> #29. Please feel free to provide further thoughts or comments about how you approach sampling. (text)

Regards,
   Shadi


> Kindest regards,
>
> Eric

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Saturday, 13 October 2012 10:33:33 UTC