Re: Input for survey about random sampling

Hi Eric,

As an evaluator I have tried to answer the questions.  Here are my comments 
regarding the questions I have problems with.

#3 Purpose of evaluation.
I had trouble with selecting three options. Most of my work is number 1 
(full evaluation), I do number 6 (presales checks - but not real 
evaluations) and I try to persuade more clients to do more number 7 
(progress evaluations). I would find it easier to complete this question if 
I were allowed to rank my choices in some way (either straight 1,2,3 etc, or 
more useful on a scale of 1 - 5 where 1 is most often and 5 is rarely)

#12 Number of pages to sample
No two sites are the same size, but nearly all require at least 100 pages to 
be sampled (checked) so this question will not help you much. It would be 
better if it were expressed as %. But even so this would depend upon the 
type of site so the question text should reference #13 and #17 which are 
going to ask about variability.

#23 Check all pages for success criteria - & #24 Report on all pages
I need to explain that for a detailed report I only record/report sufficient 
failures to illustrate an overall non-compliance whilst my in depth analysis 
lists all the failures per page.

Regards

Richard

-----Original Message----- 
From: Velleman, Eric
Sent: Saturday, October 13, 2012 2:46 PM
To: public-wai-evaltf@w3.org ; shadi@w3.org
Subject: RE: Input for survey about random sampling

Dear all,

Please find below an update of the survey questions. Please have a good 
look. I tried hard to limit the number of open questions. This should limit 
the amount of time necessary to answer the survey:


*** EVALUATION TASKFORCE CONCEPT SURVEY ABOUT SAMPLING***

The Evaluation Taskforce is working on the Website Accessibility Conformance 
Evaluation Methodology. For this methodology we need more input for the 
section on sampling. This survey contains a number of questions that will 
help us improve the section on sampling. The survey is open for EvalTF, WCAG 
and ERT working group members. We will discuss the first outcomes during the 
EvalTF group meetings at TPAC Lyon end of October 2012.

Note: You could fill in the survey more than once using different choices in 
the first and second question. Please note that the Methodology is about 
full websites (as defined in the Public Working Draft). Please read the 
Public Working Draft before you answer the questions: 
http://www.w3.org/TR/WCAG-EM/

**GENERAL**

#01. Please choose the evaluation activity for which you fill in this 
survey. If you recognize more than one of the activities, please consider 
filling out the form for each of them separately.
Your main evaluation activity is:
  - (radio) In-house website evaluation for your own organization
  - (radio) Website evaluation for external evaluation commissioner
  - (radio) Other - please specify below (text)

#02. How would you describe the goal of most of your evaluation activity 
when evaluating a full website. For an explanation of the different goals, 
please read the section about goals of an evaluation: 
http://www.w3.org/TR/WCAG-EM/#step1b
  - (checkbox) Basic evaluation
  - (checkbox) Detailed evaluation
  - (checkbox) In depth analysis

#03. For which purposes do you evaluate websites. Please choose the three 
most important for your situation:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to 
initiate a contract)
  - [checkbox] Evaluate websites in early/middle development stages
  - [checkbox] Deliver training about Web accessibility evaluation
  - (checkbox) Other - please specify below (text)

**CHOOSING THE SAMPLE**

#04. Do you select sample pages based on (note: this question is independent 
of the tools you use):
  - (radio) Structured sampling only (Selection based on certain criteria)
  - (radio) Random sampling only (Random selection)
  - (radio) Both structured and random sampling

#05. How do you select pages for a structured sample (if applicable)? (text)

#06. How do you select pages for a random sample (if applicable)? (text)

#07. How do you identify and sample the functionality (processes) provided 
on a website? (text)

#08. How do you address (mobile) web applications in your sampling? This 
includes client-side rich web applications and mobile web applications. 
(text)

#09. Do you look out for widgets, snippets, and other repeating web page 
components to avoid re-evaluating them every time they appear on a website?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

**EXTERNAL INFLUENCE ON THE CHOICE OF THE SAMPLE**

#10. Do you consult with the evaluation commissioner regarding the sample 
selection? Please note that this question is about taking a sample and not 
about setting the scope of the evaluation.
  -(radio) Always
  -(radio) Never
  -(radio) Depends on… please specify below (text)

#11. To what extent do the wishes of the evaluation commissioner influence 
the evaluation sample? Please note that the evaluation commissioner is not 
necessarily the website owner. Please read the definition of both terms at: 
<http://www.w3.org/TR/WCAG-EM/#terms>.
1= no influence (evaluator is independent in the choice of the evaluation 
sample), 5=total influence (evaluator is not independent in the choice of 
the evaluation sample)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**SIZE OF THE SAMPLE**

#12. What is the amount of pages you typically use in your sample when 
evaluating a website?
  - (radio) 1 - 5
  - (radio) 6 - 15
  - (radio) 16 - 30
  - (radio) 31 - 50
  - (radio) 51 – 100
  - (radio) More than 100

#13. If the amount of pages in your evaluation sample is variable, what 
are –in most cases – the factor(s) that have the most influence on the size 
of the sample. Please choose the three most important for your situation:
  - [checkbox] Size of the website
  - [checkbox] Number of technologies on the website
  - [checkbox] Number of templates
  - [checkbox] Variety of web page types and styles
  - [checkbox] Complexity of website functionality
  - [checkbox] Time available for the evaluation of the pages
  - [checkbox] Amount of money available for the evaluation
  - [checkbox] Other, please specify below (text)

#14. What happens to the sample when you re-evaluate a website (that has 
recently been evaluated and had failures)?
  - (radio) We make a completely new sample
  - (radio) We use the same sample as much as possible
  - (radio) We use the same sample as much as possible and add new pages to 
it
  - (radio) We only re-evaluate the pages that had failures
  - (radio) We only re-evaluate the pages that had failures plus some new 
pages
  - (radio) Don’t know

#15. Do you sample pages in different amounts for Basic Reporting, Detailed 
Reporting or In-Depth Analysis evaluation? Note that the evaluations are in 
all cases full evaluations of a website.
  - (radio) No
  - (radio) Yes, more for In-Depth Analysis
  - (radio) Yes, more for Detailed Reporting and for In-Depth Analysis
  - (radio) Don’t know

#16. How does the size of the website influence your selection?
  - (radio) It does not influence the selection
  - (radio) For larger sites, we add extra pages to the structured sample
  - (radio) For larger sites, we add extra pages to the random sample
  - (radio) For larger sites, we add extra pages to the structured and to 
the random sample
  - (radio) Other or don’t know

#17. How do you adjust your sampling approach for websites that are heavily 
template-driven (if applicable)? (text)

#18. Are the evaluation results from your current sample practice a good 
indicator of the accessibility of the overall website that you are 
evaluating?
1= No (result of evaluating the sample does not relate to rest of the 
website at all, only to the sampled pages), 5= Yes (they are a good 
indicator for the rest of the website)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**USE OF TOOLS**

#19. Do you use automated tools to help select the sample (crawlers, search 
engine etc.)?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

#20. Please specify any tools you use to help you select the sample, and how 
you use these tools for sampling. (text)

#21. Do you have a quality assurance check in place to ensure that you can 
trust the output of the tool(s) for sampling. Example: if you use a 
search-engine to find pages, do you check if the search-engine finds all the 
pages?
  - (radio) Yes,  we have a QA check for all tools we use to select the 
sample
  - (radio) Yes, we have a QA check for some tools we use to select the 
sample
  - (radio) No
  - (radio) Don’t know

**EVALUATION**

#22. You evaluate all the pages in the sample:
  - (radio) Manually
  - (radio) Automatically
  - (radio) Automatically and manually
  - (radio) Automatically and a smaller sample manually

#23. Do you evaluate all success criteria per page for all the pages in the 
sample if your goal is a detailed report or an in-depth analysis of the 
website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No
  - (radio) Don’t know

**REPORTING**

#24. Do you report all success criteria per page for all the pages in the 
sample if your goal is a detailed report or an in-depth analysis of the 
website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No

**YOUR THOUGHTS**

#25. Please feel free to provide further thoughts or comments about how you 
approach sampling. (text)


Kindest regards,

Eric

Received on Saturday, 13 October 2012 15:14:22 UTC