RE: Input for survey about random sampling

Dear EvalTF,

I missed a few earlier comments. Here is a new version with all the comments received sofar:

Concept survey questions about random sampling

#01. For which purposes do you evaluate websites:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - [checkbox] Pre-sales activities (evaluating websites in order to initiate a contract)
  - [checkbox] Evaluate websites in early/medium development stages 
  - [checkbox] Deliver training about Web accessibility evaluation 
  - (checkbox) Other - please specify below (text)

#02. Do you consult with the evaluation commissioner regarding the sample selection? 
  -(radio) Always
  -(radio) Sometimes
  -(radio) Never
  -(radio) Depend upon whether the website is being done without their knowledge

#03. Do you select sample pages for evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

#04. Please specify if you sample pages differently or in different amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation (text).

#05. How does re-evaluation of a website influence the sample? (text)

#06. Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

#07. If you use structured sampling, how do you select the individual web pages? (text)

#08. If you use random sampling, how do you select the individual web pages? (text)

#09. If you do evaluation for different purposes, how does the purpose of the evaluation influence your selection of sample pages? (text)

#10. How does the size of the website influence your selection? (text)

#11. How do you adjust your sampling approach for websites that are heavily template-driven? (text)

#12. How do you adjust your sampling approach for web applications? (text)

#13. Do you look out for widgets, snippets, and other web page components to avoid re-evaluating them every time they appear on a website? If so, how do you include them and the pages that they are integrated into the website? (text)

#14. How do you identify and sample the functionality (processes) provided on a website? (text)

#15. Do you select further sample pages based upon results from an initial selection (eg. to balance web page types or success criteria)? (text)

#16. Please feel free to provide further thoughts or comments about how you approach sampling. (text)



________________________________________
From: Velleman, Eric [evelleman@bartimeus.nl]
Sent: Thursday, 4 October 2012 9:45 PM
To: public-wai-evaltf@w3.org
Subject: RE: Input for survey about random sampling

Dear all,

Thanks to all for the comments on the list of possible questions for the group survey about random sampling. Thanks also to Shadi for the rephrasing. Please keep sending pointers to literature and articles you know of. In the meantime, this is the status. We will discuss it in our telco today:

Concept survey questions about random sampling

#01. For which purposes do you evaluate websites:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - (checkbox) Other - please specify below (text)

#02. Do you select sample pages for evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

#03. Please specify if you sample pages differently for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation (text).

#04. Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

#05. If you use structured sampling, how do you select the individual web pages? (text)

#06. If you use random sampling, how do you select the individual web pages? (text)

#07. If you do evaluation for different purposes, how does the purpose of the evaluation influence your selection of sample pages? (text)

#08. How does the size of the website influence your selection? (text)

#09. How do you adjust your sampling approach for websites that are heavily template-driven? (text)

#10. How do you adjust your sampling approach for web applications? (text)

#11. Do you look out for widgets, snippets, and other web page components to avoid re-evaluating them every time they appear on a website? If so, how do you include them and the pages that they are integrated into the website? (text)

#12. How do you identify and sample the functionality (processes) provided on a website? (text)

#13. Do you select further sample pages based upon results from an initial selection (eg. to balance web page types or success criteria)? (text)

#14. Please feel free to provide further thoughts or comments about how you approach sampling. (text)

Feel free to comment to the list.
Kindest regards,

Eric



________________________________________
Van: Boland Jr, Frederick E. [frederick.boland@nist.gov]
Verzonden: dinsdag 2 oktober 2012 17:57
Aan: Kathy Wahlbin; Shadi Abou-Zahra; Velleman, Eric
CC: public-wai-evaltf@w3.org
Onderwerp: RE: Input for survey about random sampling

Possible resource for the sampling discussion - a paper:
Effect of Sampling Methods on Web Accessibility Evaluations:
http://portal.acm.org/citation.cfm?id=1296855

May be a little bit old (from 2007) but anyway..

Thanks and best wishes
Tim Boland NIST

-----Original Message-----
From: Kathy Wahlbin [mailto:kathy@interactiveaccessibility.com]
Sent: Tuesday, October 02, 2012 10:44 AM
To: Shadi Abou-Zahra; Velleman, Eric
Cc: public-wai-evaltf@w3.org
Subject: RE: Input for survey about random sampling

Hi Eric -

I would change the second question ("Do you select sample pages for evaluation based on") slightly from what Shadi added below to break that into the different types of audits that we have in the methodology.  The approach for sample of pages would/may change based on the type of audit.

So the #2 would change to:

# Do you select sample pages for Basic Report evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

# Do you select sample pages for Detailed Report evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

# Do you select sample pages for In-Depth Analysis evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

Best,

Kathy
T (978) 443-0798  F (978) 560-1251  C (978) 760-0682 E kathyw@ia11y.com www.InteractiveAccessibility.com

NOTICE: This communication may contain privileged or other confidential information. If you are not the intended recipient, please reply to the sender indicating that fact and delete the copy you received. Thank you.

-----Original Message-----
From: Shadi Abou-Zahra [mailto:shadi@w3.org]
Sent: Tuesday, October 02, 2012 10:24 AM
To: Velleman, Eric
Cc: public-wai-evaltf@w3.org
Subject: Re: Input for survey about random sampling

Hi Eric,

Here is an attempted re-write your question for your consideration:

# For which purposes do you evaluate websites:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - (checkbox) Other - please specify below (text)

# Do you select sample pages for evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

# Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

# If you use structured sampling, how do you select the individual web pages? (text)

# If you use random sampling, how do you select the individual web pages? (text)

# If you do evaluation for different purposes, how does the purpose of the evaluation influence your selection of sample pages? (text)

# How does the size of the website influence your selection? (text)

# How do you adjust your sampling approach for websites that are heavily template-driven? (text)

# How do you adjust your sampling approach for web applications? (text)

# Do you look out for widgets, snippets, and other web page components to avoid re-evaluating them every time they appear on a website? If so, how do you include them and the pages that they are integrated into the website? (text)

# How do you identify and sample the functionality (processes) provided on a website? (text)

# Do you select further sample pages based upon results from an initial selection (eg. to balance web page types or success criteria)? (text)

# Please feel free to provide further thoughts or comments about how you approach sampling. (text)


I wonder if we need to break some of these down even more so that we are asking ourselves very specific questions? Or maybe that will be a future iteration where we want to find out more details? Not sure.

Best,
   Shadi


On 2.10.2012 15:02, Velleman, Eric wrote:
> Dear EvalTF,
>
> In our last telco we discussed opening a group only survey about random sampling. This would then provide input to draft a first text.
>
> Please add questions to the list below that may be relevant to draft a first text about random sampling. Here are a few from the last telco and my own additions. Please mark them with YES or NO to delete or keep and add questions as you like:
>
> - How do you collect pages?
> - How do you collect pages in web-applications?
> - Is there a difference between large and small websites in the manner you collect the pages?
> - Is there a difference between large and small websites in the size of the sample?
> - Do you collect a random sample?
> - Do you think random sampling is necessary? and why?
> - What portion of your sample is randomly selected?
> - How do you collect randomly (do you count pages..)?
> - If you use tools for sampling?
> - Are tools sufficient to collect a sample?
> - Should the random sample be other pages than the ones already included in Step 3?
> - ...
> - ...
> - ...
>
> Sampling is now in our methodology in section 3.3 Step 3 Select a Representative Sample. The current version of the methodology lets you select a representative sample including Common Web Pages of the Website, Exemplar Instances of Web Pages, Other Relevant Web Pages and Complete Processes. Random sampling is not included.
>
> The endresult for this survey would be about 10 to 15 questions.
> We could set the max for answers to 100 words per question.
> Kindest regards,
>
> Eric
>
>
>

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program Office Evaluation and Repair Tools Working Group (ERT WG) Research and Development Working Group (RDWG)

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B

Received on Thursday, 4 October 2012 13:59:57 UTC