RE: Input for survey about random sampling

Dear all,

Find below a new version of the survey. Will work on some remarks by Richard when we implement the survey in a tool. They seem to depend on the tool we use.

*** EVALUATION TASKFORCE CONCEPT SURVEY ABOUT SAMPLING***

The Evaluation Taskforce is working on the Website Accessibility Conformance Evaluation Methodology. For this methodology we need more input for the section on sampling. This survey contains a number of questions that will help us improve the section on sampling. The survey is open for EvalTF, WCAG and ERT working group members. We will discuss the first outcomes during the EvalTF group meetings at TPAC Lyon end of October 2012.

Note: You could fill in the survey more than once using different choices in the first and second question. Please note that the Methodology is about full websites (as defined in the Public Working Draft). Please read the Public Working Draft before you answer the questions: http://www.w3.org/TR/WCAG-EM/

**GENERAL**

#01. Please choose the evaluation activity for which you fill in this survey. If you recognize more than one of the activities, please consider filling out the form for each of them separately.
Your main evaluation activity is: 
  - (radio) In-house website evaluation for your own organization
  - (radio) Website evaluation for external evaluation commissioner
  - (radio) Other - please specify below (text)

#02. How would you describe the goal of most of your evaluation activity when evaluating a full website. For an explanation of the different goals, please read the section about goals of an evaluation: http://www.w3.org/TR/WCAG-EM/#step1b
  - (checkbox) Basic evaluation 
  - (checkbox) Detailed evaluation
  - (checkbox) In depth analysis  

#03. For which purposes do you mainly evaluate websites? Please choose the three most important for your situation:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to initiate a contract)
  - [checkbox] Evaluate websites in early/middle development stages 
  - [checkbox] Deliver training about Web accessibility evaluation 
  - (checkbox) Other - please specify below (text)

**Editor Note: could we do a ranking of question #03?


**CHOOSING THE SAMPLE**


#04. Do you or your organization follow a strict methodology for the selection of the sample?
  - (radio) Yes
  - (radio) No

#05. Do you select sample pages based on (note: this question is independent of the tools you use):
  - (radio) Structured sampling only (Selection based on certain criteria)
  - (radio) Random sampling only (Random selection)
  - (radio) Both structured and random sampling

#06. How do you select pages for a structured sample (if applicable)? (text)

#07. How do you select pages for a random sample (if applicable)? (text)

#08. How do you identify and sample the functionality (processes) provided on a website? (text)

#09. How do you address (mobile) web applications in your sampling? This includes client-side rich web applications and mobile web applications. (text) 

#10. Do you look out for widgets, snippets, and other repeating web page components to avoid re-evaluating them every time they appear on a website? 
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

**EXTERNAL INFLUENCE ON THE CHOICE OF THE SAMPLE**

#11. Do you consult with the evaluation commissioner regarding the sample selection? Please note that this question is about taking a sample and not about setting the scope of the evaluation. 
  -(radio) Always
  -(radio) Never
  -(radio) Depends on… please specify below (text)

#12. To what extent do the wishes of the evaluation commissioner influence your  evaluation sample? Please note that the evaluation commissioner is not necessarily the website owner. Please read the definition of both terms at: <http://www.w3.org/TR/WCAG-EM/#terms>. 
1= no influence (evaluator is independent in the choice of the evaluation sample), 5=total influence (evaluator is not independent in the choice of the evaluation sample)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**SIZE OF THE SAMPLE**

#13. What is the amount of pages you typically use in your sample when evaluating a website? (Please note that questions 13 and 17 will further address variability)
  - (radio) 1 - 5
  - (radio) 6 - 10
  - (radio) 11 - 25
  - (radio) 26 - 50
  - (radio) 51 – 100
  - (radio) More than 100
  - (radio) We use a percentage, please answer question below

#14. If you use a percentage, what percentage would describe the amount of pages that you typically use in your sample when evaluating a website? (text)

#15. If the amount of pages in your evaluation sample is variable, what are –in most cases – the factor(s) that have the most influence on the size of the sample. Please choose the three most important for your situation:
  - [checkbox] Size of the website
  - [checkbox] Number of technologies on the website
  - [checkbox] Number of templates
  - [checkbox] Variety of web page types and styles
  - [checkbox] Complexity of website functionality
  - [checkbox] Time available for the evaluation of the pages
  - [checkbox] Amount of money available for the evaluation
  - [checkbox] Other, please specify below (text)

#16. What happens to the sample when you re-evaluate a website (that has recently been evaluated and had failures)?
  - (radio) We make a completely new sample 
  - (radio) We use the same sample as much as possible
  - (radio) We use the same sample as much as possible and add new pages to it
  - (radio) We only re-evaluate the pages that had failures
  - (radio) We only re-evaluate the pages that had failures plus some new pages
  - (radio) Don’t know

#17. Do you sample pages in different amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation? Note that the evaluations are in all cases full evaluations of a website.  
  - (radio) No
  - (radio) Yes, more for In-Depth Analysis
  - (radio) Yes, more for Detailed Reporting and for In-Depth Analysis
  - (radio) Don’t know

#18. How does the size of the website influence your selection?
  - (radio) It does not influence the selection
  - (radio) For larger sites, we add extra pages to the structured sample
  - (radio) For larger sites, we add extra pages to the random sample
  - (radio) For larger sites, we add extra pages to the structured and to the random sample
  - (radio) Other or don’t know

#19. How do you adjust your sampling approach for websites that are heavily template-driven (if applicable)? (text)

#20. Are the evaluation results from your current sample practice a good indicator of the accessibility of the overall website that you are evaluating? 
1= No (result of evaluating the sample does not relate to rest of the website at all, only to the sampled pages), 5= Yes (they are a good indicator for the rest of the website)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**USE OF TOOLS**

#21. Do you use automated tools to help select the sample (crawlers, search engine etc.)?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

#22. Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

#23. Do you have a quality assurance check in place to ensure that you can trust the output of the tool(s) for sampling. Example: if you use a search-engine to find pages, do you check if the search-engine finds all the pages?
  - (radio) Yes,  we have a QA check for all tools we use to select the sample
  - (radio) Yes, we have a QA check for some tools we use to select the sample
  - (radio) No
  - (radio) Don’t know

**EVALUATION**

#24. You evaluate all the pages in the sample:
  - (radio) Manually
  - (radio) Automatically 
  - (radio) Automatically and manually
  - (radio) Automatically and a smaller sample manually

#25. Do you evaluate all success criteria per page for all the pages in the sample if your goal is a detailed report or an in-depth analysis of the website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No
  - (radio) Other, please clarify. (text)

#26. Are you confident that another evaluator with your expertise and experience would come to the same evaluation results for a given website?
  - (radio) Yes
  - (radio) No

**REPORTING**

#27. Do you report all success criteria per page for all the pages in the sample if your goal is a detailed report or an in-depth analysis of the website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No
  - (radio) Other, please clarify. (text)

**YOUR THOUGHTS**

#28. Please feel free to provide further thoughts or comments about how you approach sampling. (text)


Kindest regards,

Eric

=========================
Eric Velleman
Technisch directeur
Stichting Accessibility
Universiteit Twente

Tel: +31 (0)30 - 2398270

Christiaan Krammlaan 2
3571 AX Utrecht

www.accessibility.nl / www.wabcluster.org / www.econformance.eu /
www.game-accessibility.com/ www.eaccessplus.eu

Lees onze disclaimer: www.accessibility.nl/algemeen/disclaimer
Accessibility is Member van het W3C
=========================

________________________________________
Van: Velleman, Eric
Verzonden: zaterdag 13 oktober 2012 17:38
Aan: RichardWarren; public-wai-evaltf@w3.org; shadi@w3.org
Onderwerp: RE: Input for survey about random sampling

Hi Richard,

Thank you, seem like easy to do options and changes. For the ranking, it will depend a bit on the possibilities of the survey tool but I agree that it would be good to have that.
Kindest regards,

Eric



________________________________________
Van: RichardWarren [richard.warren@userite.com]
Verzonden: zaterdag 13 oktober 2012 17:13
Aan: Velleman, Eric; public-wai-evaltf@w3.org; shadi@w3.org
Onderwerp: Re: Input for survey about random sampling

Hi Eric,

As an evaluator I have tried to answer the questions.  Here are my comments
regarding the questions I have problems with.

#3 Purpose of evaluation.
I had trouble with selecting three options. Most of my work is number 1
(full evaluation), I do number 6 (presales checks - but not real
evaluations) and I try to persuade more clients to do more number 7
(progress evaluations). I would find it easier to complete this question if
I were allowed to rank my choices in some way (either straight 1,2,3 etc, or
more useful on a scale of 1 - 5 where 1 is most often and 5 is rarely)

#12 Number of pages to sample
No two sites are the same size, but nearly all require at least 100 pages to
be sampled (checked) so this question will not help you much. It would be
better if it were expressed as %. But even so this would depend upon the
type of site so the question text should reference #13 and #17 which are
going to ask about variability.

#23 Check all pages for success criteria - & #24 Report on all pages
I need to explain that for a detailed report I only record/report sufficient
failures to illustrate an overall non-compliance whilst my in depth analysis
lists all the failures per page.

Regards

Richard

-----Original Message-----
From: Velleman, Eric
Sent: Saturday, October 13, 2012 2:46 PM
To: public-wai-evaltf@w3.org ; shadi@w3.org
Subject: RE: Input for survey about random sampling

Dear all,

Please find below an update of the survey questions. Please have a good
look. I tried hard to limit the number of open questions. This should limit
the amount of time necessary to answer the survey:


*** EVALUATION TASKFORCE CONCEPT SURVEY ABOUT SAMPLING***

The Evaluation Taskforce is working on the Website Accessibility Conformance
Evaluation Methodology. For this methodology we need more input for the
section on sampling. This survey contains a number of questions that will
help us improve the section on sampling. The survey is open for EvalTF, WCAG
and ERT working group members. We will discuss the first outcomes during the
EvalTF group meetings at TPAC Lyon end of October 2012.

Note: You could fill in the survey more than once using different choices in
the first and second question. Please note that the Methodology is about
full websites (as defined in the Public Working Draft). Please read the
Public Working Draft before you answer the questions:
http://www.w3.org/TR/WCAG-EM/

**GENERAL**

#01. Please choose the evaluation activity for which you fill in this
survey. If you recognize more than one of the activities, please consider
filling out the form for each of them separately.
Your main evaluation activity is:
  - (radio) In-house website evaluation for your own organization
  - (radio) Website evaluation for external evaluation commissioner
  - (radio) Other - please specify below (text)

#02. How would you describe the goal of most of your evaluation activity
when evaluating a full website. For an explanation of the different goals,
please read the section about goals of an evaluation:
http://www.w3.org/TR/WCAG-EM/#step1b
  - (checkbox) Basic evaluation
  - (checkbox) Detailed evaluation
  - (checkbox) In depth analysis

#03. For which purposes do you evaluate websites. Please choose the three
most important for your situation:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to
initiate a contract)
  - [checkbox] Evaluate websites in early/middle development stages
  - [checkbox] Deliver training about Web accessibility evaluation
  - (checkbox) Other - please specify below (text)

**CHOOSING THE SAMPLE**

#04. Do you select sample pages based on (note: this question is independent
of the tools you use):
  - (radio) Structured sampling only (Selection based on certain criteria)
  - (radio) Random sampling only (Random selection)
  - (radio) Both structured and random sampling

#05. How do you select pages for a structured sample (if applicable)? (text)

#06. How do you select pages for a random sample (if applicable)? (text)

#07. How do you identify and sample the functionality (processes) provided
on a website? (text)

#08. How do you address (mobile) web applications in your sampling? This
includes client-side rich web applications and mobile web applications.
(text)

#09. Do you look out for widgets, snippets, and other repeating web page
components to avoid re-evaluating them every time they appear on a website?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

**EXTERNAL INFLUENCE ON THE CHOICE OF THE SAMPLE**

#10. Do you consult with the evaluation commissioner regarding the sample
selection? Please note that this question is about taking a sample and not
about setting the scope of the evaluation.
  -(radio) Always
  -(radio) Never
  -(radio) Depends on… please specify below (text)

#11. To what extent do the wishes of the evaluation commissioner influence
the evaluation sample? Please note that the evaluation commissioner is not
necessarily the website owner. Please read the definition of both terms at:
<http://www.w3.org/TR/WCAG-EM/#terms>.
1= no influence (evaluator is independent in the choice of the evaluation
sample), 5=total influence (evaluator is not independent in the choice of
the evaluation sample)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**SIZE OF THE SAMPLE**

#12. What is the amount of pages you typically use in your sample when
evaluating a website?
  - (radio) 1 - 5
  - (radio) 6 - 15
  - (radio) 16 - 30
  - (radio) 31 - 50
  - (radio) 51 – 100
  - (radio) More than 100

#13. If the amount of pages in your evaluation sample is variable, what
are –in most cases – the factor(s) that have the most influence on the size
of the sample. Please choose the three most important for your situation:
  - [checkbox] Size of the website
  - [checkbox] Number of technologies on the website
  - [checkbox] Number of templates
  - [checkbox] Variety of web page types and styles
  - [checkbox] Complexity of website functionality
  - [checkbox] Time available for the evaluation of the pages
  - [checkbox] Amount of money available for the evaluation
  - [checkbox] Other, please specify below (text)

#14. What happens to the sample when you re-evaluate a website (that has
recently been evaluated and had failures)?
  - (radio) We make a completely new sample
  - (radio) We use the same sample as much as possible
  - (radio) We use the same sample as much as possible and add new pages to
it
  - (radio) We only re-evaluate the pages that had failures
  - (radio) We only re-evaluate the pages that had failures plus some new
pages
  - (radio) Don’t know

#15. Do you sample pages in different amounts for Basic Reporting, Detailed
Reporting or In-Depth Analysis evaluation? Note that the evaluations are in
all cases full evaluations of a website.
  - (radio) No
  - (radio) Yes, more for In-Depth Analysis
  - (radio) Yes, more for Detailed Reporting and for In-Depth Analysis
  - (radio) Don’t know

#16. How does the size of the website influence your selection?
  - (radio) It does not influence the selection
  - (radio) For larger sites, we add extra pages to the structured sample
  - (radio) For larger sites, we add extra pages to the random sample
  - (radio) For larger sites, we add extra pages to the structured and to
the random sample
  - (radio) Other or don’t know

#17. How do you adjust your sampling approach for websites that are heavily
template-driven (if applicable)? (text)

#18. Are the evaluation results from your current sample practice a good
indicator of the accessibility of the overall website that you are
evaluating?
1= No (result of evaluating the sample does not relate to rest of the
website at all, only to the sampled pages), 5= Yes (they are a good
indicator for the rest of the website)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

**USE OF TOOLS**

#19. Do you use automated tools to help select the sample (crawlers, search
engine etc.)?
  - (radio) Yes
  - (radio) No
  - (radio) Don’t know

#20. Please specify any tools you use to help you select the sample, and how
you use these tools for sampling. (text)

#21. Do you have a quality assurance check in place to ensure that you can
trust the output of the tool(s) for sampling. Example: if you use a
search-engine to find pages, do you check if the search-engine finds all the
pages?
  - (radio) Yes,  we have a QA check for all tools we use to select the
sample
  - (radio) Yes, we have a QA check for some tools we use to select the
sample
  - (radio) No
  - (radio) Don’t know

**EVALUATION**

#22. You evaluate all the pages in the sample:
  - (radio) Manually
  - (radio) Automatically
  - (radio) Automatically and manually
  - (radio) Automatically and a smaller sample manually

#23. Do you evaluate all success criteria per page for all the pages in the
sample if your goal is a detailed report or an in-depth analysis of the
website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No
  - (radio) Don’t know

**REPORTING**

#24. Do you report all success criteria per page for all the pages in the
sample if your goal is a detailed report or an in-depth analysis of the
website?
  - (radio) Yes
  - (radio) Yes, but only for in-depth analysis
  - (radio) No

**YOUR THOUGHTS**

#25. Please feel free to provide further thoughts or comments about how you
approach sampling. (text)


Kindest regards,

Eric

Received on Monday, 15 October 2012 12:25:07 UTC