RE: Input for survey about random sampling

Hi Eric, 

Some feedback on the survey questions:

#02. For which purposes do you evaluate websites:
 - [checkbox] Evaluate websites in early/medium development stages - 
Change medium to middle. 

#03. Do you consult with the evaluation commissioner regarding the sample 
selection?
 -(radio) Depend upon whether the website is being done without their 
knowledge -  I would recommend changing to:  Depends upon whether the 
website is being evaluated with or without the knowledge of the evaluation 
commissioner.

#05. Do you select sample pages for evaluation based on:
How about Structure plus automated sampling?

#12. What is the amount of pages you use in your sample when evaluating a 
website?
Needed to have a choice for greater than 100 especially if automated 
tooling is used. Maybe the choices should be 1-10, 11-50, 51-100, more 
than 100 pages.

#13. If the amount of pages in your evaluation sample is variable, what 
are –in most cases – the factor(s) that have the most influence on the 
size of the sample:
Another option may be due to number of new features or content since last 
evaluation.

Talk to you shortly. 

Thanks,
Moe



Maureen Kraft
Accessibility Consultant Test Lead
Human Ability and Accessibility Center 
IBM Research 

Tel: 978-899-3114
E-mail: maureen_kraft@us.ibm.com 


 

















From:   "Velleman, Eric" <evelleman@bartimeus.nl>
To:     "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>, 
Date:   10/10/2012 05:24 PM
Subject:        RE: Input for survey about random sampling



Dear EvalTF,

Following our discussion last telco, the minutes of that meeting and some 
remarks sent to me, this is an update of the survey. Please send your 
comments to the list. Before we start making this a scientific research 
project, please keep in mind that this is input information for a first 
draft text to add a random sample.

** Concept survey questions about sampling **

Note: You could fill in the survey more than once using different choices 
in the first question. Please note that the Methodology is about full 
websites (as defined in the Public Working Draft).

#01. Please choose the situation below for which you fill in this survey. 
Your evaluation activity is: 
  - (checkbox) In-house website evaluation for your own organization
  - (checkbox) Website evaluation for external evaluation commissioner
  - (checkbox) A re-evaluation of a website that you have evaluated less 
than 1 month ago
  - (checkbox) In depth evaluation
  - (checkbox) Basic evaluation
  - (checkbox) Detailed evaluation 
  - (checkbox) Other - please specify below (text)

Please fill in the questions depending on the evaluation activity you have 
chosen in question #01:

#02. For which purposes do you evaluate websites:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to 
initiate a contract)
  - [checkbox] Evaluate websites in early/medium development stages 
  - [checkbox] Deliver training about Web accessibility evaluation 
  - (checkbox) Other - please specify below (text)

#03. Do you consult with the evaluation commissioner regarding the sample 
selection? 
  -(radio) Always
  -(radio) Sometimes
  -(radio) Never
  -(radio) Depend upon whether the website is being done without their 
knowledge

#04. To what extent do the wishes of the evaluation commissioner influence 
the evaluation sample? 1= no influence (evaluator is independent in choice 
of evaluation sample), 5=total influence (evaluator is not independent in 
choice of of evaluation sample)
  -(radio) 1
  -(radio) 2
  -(radio) 3
  -(radio) 4
  -(radio) 5

#05. Do you select sample pages for evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Automated sampling only
  - (radio) Both structured and random sampling
  - (radio) Both structured and random sampling supported by automated 
sampling

#06. Please specify any tools you use to help you select the sample, and 
how you use these tools for sampling. (text)

#07. If you use automated sampling only, how do you check the tool? (text)

#08. If you use structured sampling, how do you select the individual web 
pages? (text)

#09. If you use random sampling, how do you select the individual web 
pages? (text)

#10. How does re-evaluation of a website influence the sample? (text)

#11. Please specify if you sample pages differently or in different 
amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis 
evaluation (text). 

#12. What is the amount of pages you use in your sample when evaluating a 
website?
  - (radio) 1 - 5
  - (radio) 6 - 15
  - (radio) 16 - 30
  - (radio) 31 - 50
  - (radio) 51 - 100

#13. If the amount of pages in your evaluation sample is variable, what 
are –in most cases – the factor(s) that have the most influence on the 
size of the sample:
  - [checkbox] Size of the website
  - [checkbox] Number of technologies on the website
  - [checkbox] Amount of money available for the evaluation
  - [checkbox] Number of templates
  - [checkbox] Other, please specify below (text)

#14. How does the size of the website influence your selection? (text)

#15. How do you adjust your sampling approach for websites that are 
heavily template-driven? (text)

#16. If you use automated testing tool(s), does that change your sampling 
methodology?
  - (radio) Yes
  - (radio) No
  - Please explain (text)

#17. How do you adjust your sampling approach for web applications? (text)

#18. How do you sample mobile applications? (text)

#19. Do you look out for widgets, snippets, and other web page components 
to avoid re-evaluating them every time they appear on a website? 
  - (radio) Yes
  - (radio) No
If Yes, how do you include them and the pages that they are integrated 
into the sample? (text)

#20. How do you identify and sample the functionality (processes) provided 
on a website? (text)

#21. Do you select further sample pages based upon results from an initial 
selection (eg. to balance web page types or success criteria)? (text)

#22. Please feel free to provide further thoughts or comments about how 
you approach sampling. (text)

Kindest regards,

Eric