RE: Input for survey about random sampling

Hi Eric & TF

I like the way this is looking. Kathy's suggestion for breaking down question #2 is also very appropriate.

I think we should add a question as to the role played by the evaluation commissioner in the page selection.

Do you consult with the evaluation commissioner regarding the sample selection? (always/sometimes/never/other - may depend upon whether the website is being done without their knowledge)
What role does the evaluation commissioner play in the sample selection? (text)




Regards

Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
________________________________________
From: Shadi Abou-Zahra [shadi@w3.org]
Sent: Tuesday, 2 October 2012 10:23 PM
To: Velleman, Eric
Cc: public-wai-evaltf@w3.org
Subject: Re: Input for survey about random sampling

Hi Eric,

Here is an attempted re-write your question for your consideration:

# For which purposes do you evaluate websites:
  - (checkbox) Provide in-depth analyses to guide website owners
  - (checkbox) Monitor the accessibility of a website over time
  - (checkbox) Final check before releasing or purchasing a website
  - (checkbox) Provide label or other certificate of conformance
  - (checkbox) Do large-scale evaluation of many websites
  - (checkbox) Other - please specify below (text)

# Do you select sample pages for evaluation based on:
  - (radio) Structured sampling only
  - (radio) Random sampling only
  - (radio) Both structured and random sampling

# Please specify any tools you use to help you select the sample, and
how you use these tools for sampling. (text)

# If you use structured sampling, how do you select the individual web
pages? (text)

# If you use random sampling, how do you select the individual web
pages? (text)

# If you do evaluation for different purposes, how does the purpose of
the evaluation influence your selection of sample pages? (text)

# How does the size of the website influence your selection? (text)

# How do you adjust your sampling approach for websites that are heavily
template-driven? (text)

# How do you adjust your sampling approach for web applications? (text)

# Do you look out for widgets, snippets, and other web page components
to avoid re-evaluating them every time they appear on a website? If so,
how do you include them and the pages that they are integrated into the
website? (text)

# How do you identify and sample the functionality (processes) provided
on a website? (text)

# Do you select further sample pages based upon results from an initial
selection (eg. to balance web page types or success criteria)? (text)

# Please feel free to provide further thoughts or comments about how you
approach sampling. (text)


I wonder if we need to break some of these down even more so that we are
asking ourselves very specific questions? Or maybe that will be a future
iteration where we want to find out more details? Not sure.

Best,
   Shadi


On 2.10.2012 15:02, Velleman, Eric wrote:
> Dear EvalTF,
>
> In our last telco we discussed opening a group only survey about random sampling. This would then provide input to draft a first text.
>
> Please add questions to the list below that may be relevant to draft a first text about random sampling. Here are a few from the last telco and my own additions. Please mark them with YES or NO to delete or keep and add questions as you like:
>
> - How do you collect pages?
> - How do you collect pages in web-applications?
> - Is there a difference between large and small websites in the manner you collect the pages?
> - Is there a difference between large and small websites in the size of the sample?
> - Do you collect a random sample?
> - Do you think random sampling is necessary? and why?
> - What portion of your sample is randomly selected?
> - How do you collect randomly (do you count pages..)?
> - If you use tools for sampling?
> - Are tools sufficient to collect a sample?
> - Should the random sample be other pages than the ones already included in Step 3?
> - ...
> - ...
> - ...
>
> Sampling is now in our methodology in section 3.3 Step 3 Select a Representative Sample. The current version of the methodology lets you select a representative sample including Common Web Pages of the Website, Exemplar Instances of Web Pages, Other Relevant Web Pages and Complete Processes. Random sampling is not included.
>
> The endresult for this survey would be about 10 to 15 questions.
> We could set the max for answers to 100 words per question.
> Kindest regards,
>
> Eric
>
>
>

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B

Received on Wednesday, 3 October 2012 00:19:05 UTC