RE: Input for survey about random sampling

Dear all,

Please find below an update of the survey questions. Please have a good look. I tried hard to limit the number of open questions. This should limit the amount of time necessary to answer the survey:


*** EVALUATION TASKFORCE CONCEPT SURVEY ABOUT SAMPLING***

The Evaluation Taskforce is working on the Website Accessibility Conformance Evaluation Methodology. For this methodology we need more input for the section on sampling. This survey contains a number of questions that will help us improve the section on sampling. The survey is open for EvalTF, WCAG and ERT working group members. We will discuss the first outcomes during the EvalTF group meetings at TPAC Lyon end of October 2012.

Note: You could fill in the survey more than once using different choices in the first and second question. Please note that the Methodology is about full websites (as defined in the Public Working Draft). Please read the Public Working Draft before you answer the questions: http://www.w3.org/TR/WCAG-EM/

**GENERAL**

#01. Please choose the evaluation activity for which you fill in this survey. If you recognize more than one of the activities, please consider filling out the form for each of them separately.
Your main evaluation activity is: 
  - (radio) In-house website evaluation for your own organization
  - (radio) Website evaluation for external evaluation commissioner
  - (radio) Other - please specify below (text)

#02. How would you describe the goal of most of your evaluation activity when evaluating a full website. For an explanation of the different goals, please read the section about goals of an evaluation: http://www.w3.org/TR/WCAG-EM/#step1b
  - (checkbox) Basic evaluation 
  - (checkbox) Detailed evaluation
  - (checkbox) In depth analysis  

#03. For which purposes do you evaluate websites. Please choose the three most important for your situation:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to initiate a contract)
  - [checkbox] Evaluate websites in early/middle development stages 
  - [checkbox] Deliver training about Web accessibility evaluation 
  - (checkbox) Other - please specify below (text)

**CHOOSING THE SAMPLE**

#04. Do you select sample pages based on (note: this question is independent of the tools you use):
  - (radio) Structured sampling only (Selection based on certain criteria)
  - (radio) Random sampling only (Random selection)
  - (radio) Both structured and random sampling

#05. How do you select pages for a structured sample (if applicable)? (text)

#06. How do you select pages for a random sample (if applicable)? (text)

#07. How do you identify and sample the functionality (processes) provided on a website? (text)

#08. How do you address (mobile) web applications in your sampling? This includes client-side rich web applications and mobile web applications. (text) 

#09. Do you look out for widgets, snippets, and other repeating web page components to avoid re-evaluating them every time they appear on a website? 
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

**EXTERNAL INFLUENCE ON THE CHOICE OF THE SAMPLE**

#10. Do you consult with the evaluation commissioner regarding the sample selection? Please note that this question is about taking a sample and not about setting the scope of the evaluation. 
  -(radio) Always
  -(radio) Never
  -(radio) Depends on… please specify below (text)

#11. To what extent do the wishes of the evaluation commissioner influence the evaluation sample? Please note that the evaluation commissioner is not necessarily the website owner. Please read the definition of both terms at: <http://www.w3.org/TR/WCAG-EM/#terms>. 
1= no influence (evaluator is independent in the choice of the evaluation sample), 5=total influence (evaluator is not independent in the choice of the evaluation sample)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**SIZE OF THE SAMPLE**

#12. What is the amount of pages you typically use in your sample when evaluating a website?
  - (radio) 1 - 5
  - (radio) 6 - 15
  - (radio) 16 - 30
  - (radio) 31 - 50
  - (radio) 51 – 100
  - (radio) More than 100

#13. If the amount of pages in your evaluation sample is variable, what are –in most cases – the factor(s) that have the most influence on the size of the sample. Please choose the three most important for your situation:
  - [checkbox] Size of the website
  - [checkbox] Number of technologies on the website
  - [checkbox] Number of templates
  - [checkbox] Variety of web page types and styles
  - [checkbox] Complexity of website functionality
  - [checkbox] Time available for the evaluation of the pages
  - [checkbox] Amount of money available for the evaluation
  - [checkbox] Other, please specify below (text)

#14. What happens to the sample when you re-evaluate a website (that has recently been evaluated and had failures)?
  - (radio) We make a completely new sample 
  - (radio) We use the same sample as much as possible
  - (radio) We use the same sample as much as possible and add new pages to it
  - (radio) We only re-evaluate the pages that had failures
  - (radio) We only re-evaluate the pages that had failures plus some new pages
  - (radio) Don’t know

#15. Do you sample pages in different amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation? Note that the evaluations are in all cases full evaluations of a website.  
  - (radio) No
  - (radio) Yes, more for In-Depth Analysis
  - (radio) Yes, more for Detailed Reporting and for In-Depth Analysis
  - (radio) Don’t know

#16. How does the size of the website influence your selection?
  - (radio) It does not influence the selection
  - (radio) For larger sites, we add extra pages to the structured sample
  - (radio) For larger sites, we add extra pages to the random sample
  - (radio) For larger sites, we add extra pages to the structured and to the random sample
  - (radio) Other or don’t know

#17. How do you adjust your sampling approach for websites that are heavily template-driven (if applicable)? (text)

#18. Are the evaluation results from your current sample practice a good indicator of the accessibility of the overall website that you are evaluating? 
1= No (result of evaluating the sample does not relate to rest of the website at all, only to the sampled pages), 5= Yes (they are a good indicator for the rest of the website)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**USE OF TOOLS**

#19. Do you use automated tools to help select the sample (crawlers, search engine etc.)?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

#20. Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

#21. Do you have a quality assurance check in place to ensure that you can trust the output of the tool(s) for sampling. Example: if you use a search-engine to find pages, do you check if the search-engine finds all the pages?
  - (radio) Yes,  we have a QA check for all tools we use to select the sample
  - (radio) Yes, we have a QA check for some tools we use to select the sample
  - (radio) No
  - (radio) Don’t know

**EVALUATION**

#22. You evaluate all the pages in the sample:
  - (radio) Manually
  - (radio) Automatically 
  - (radio) Automatically and manually
  - (radio) Automatically and a smaller sample manually

#23. Do you evaluate all success criteria per page for all the pages in the sample if your goal is a detailed report or an in-depth analysis of the website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No
  - (radio) Don’t know

**REPORTING**

#24. Do you report all success criteria per page for all the pages in the sample if your goal is a detailed report or an in-depth analysis of the website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No

**YOUR THOUGHTS**

#25. Please feel free to provide further thoughts or comments about how you approach sampling. (text)


Kindest regards,

Eric

Received on Saturday, 13 October 2012 13:47:21 UTC