RE: Input for survey about random sampling

Hi All,

Think it also depends on the way we select the structured sample. For some pages, that is not random (homepage, searchpage,..), but I could imagine that finding a page with forms on it or a page with video could be rather random. Thus supporting the point made by Michael earlier that we need to define what random is...

Looking at my own situation, I would just look around on the website until I find a page with forms or with video.. Is that random?
Kindest regards,

Eric

=========================
Eric Velleman
Technisch directeur
Stichting Accessibility
Universiteit Twente

Tel: +31 (0)30 - 2398270

Christiaan Krammlaan 2
3571 AX Utrecht

www.accessibility.nl / www.wabcluster.org / www.econformance.eu /
www.game-accessibility.com/ www.eaccessplus.eu

Lees onze disclaimer: www.accessibility.nl/algemeen/disclaimer
Accessibility is Member van het W3C
=========================
________________________________
Van: Vivienne CONWAY [v.conway@ecu.edu.au]
Verzonden: donderdag 11 oktober 2012 15:38
Aan: Moe Kraft; Velleman, Eric
CC: public-wai-evaltf@w3.org
Onderwerp: *** Detected as Phishing *** RE: Input for survey about random sampling

Sorry Moe, perhaps I didn't explain this very well.  What the WCAG group said was that they'd like to see us recommend that 25% of the sample size be randomly chosen pages.  So if you're selecting 20 pages for your sample, then 5 should be randomly chosen if you followed this idea.  This was just a suggestion from someone in the WCAG group that reviewed our methodology.


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au<mailto:v.conway@ecu.edu.au>
v.conway@webkeyit.com<mailto:v.conway@webkeyit.com>
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.

________________________________
From: Moe Kraft [maureen_kraft@us.ibm.com]
Sent: Thursday, 11 October 2012 8:57 PM
To: evelleman@bartimeus.nl
Cc: public-wai-evaltf@w3.org; Vivienne CONWAY
Subject: RE: Input for survey about random sampling

I have to agree with Eric. It is difficult to put a percentage on large sites or applications. My team has recently been reviewing a large retail site with over 10,000 pages. The code is pretty static and the failures found are repetitive page to page. It is the content that is changing. It would be difficult to test 25% of this application which would result in 2500 pages.

Where this is a recommendation, I'm not sure we can quantify a percentage for the random sample. If we were, maybe it should be a percentage of the relative sample. Also, lets not forget about applications where the relative sample is really taking a look at user tasks; creating, searching, reading and editing documents or performing a basic task. The random sampling will differ site to site, application to application.

Moe
________________________________

[cid:_2_0A05A0CC0A059CF800472F6D85257A94]<http://www.ibm.com/able>      Maureen Kraft
Accessibility Consultant Test Lead
Human Ability and Accessibility Center
IBM Research

Tel: 978-899-3114
E-mail: maureen_kraft@us.ibm.com
                        [cid:_1_0F4BB8C40F4BB4F000472F6D85257A94]

<http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>    <http://www.facebook.com/IBMAccessibility>      <http://twitter.com/IBMAccess>






From:        "Velleman, Eric" <evelleman@bartimeus.nl>
To:        Vivienne CONWAY <v.conway@ecu.edu.au>, "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>,
Date:        10/11/2012 02:53 AM
Subject:        RE: Input for survey about random sampling
________________________________



Hi Vivienne,

Maybe we should also add a question about how people conclude the number of pages of a website.
Personally I would be interested to know the exact number of pages people use. For the BITVtest in Germany it seems to be 3 - 5 pages, for other testing schemes it is mostly not more than 30 - 50 pages. If we would take 20 percent of a website of 1.000.000 pages, that would mean checking many pages. In our discussion, there seems to be a tendency towards less pages.

This is also something to discuss in the Telco when we talk about confidence. How many pages do you need to have a certain amount of confidence in the results.

Kindest regards,

Eric

=========================
Eric Velleman
Technisch directeur
Stichting Accessibility
Universiteit Twente

Tel: +31 (0)30 - 2398270

Christiaan Krammlaan 2
3571 AX Utrecht

www.accessibility.nl<UrlBlockedError.aspx> / www.wabcluster.org<UrlBlockedError.aspx> / www.econformance.eu<UrlBlockedError.aspx> /
www.game-accessibility.com/<UrlBlockedError.aspx> www.eaccessplus.eu<UrlBlockedError.aspx>

Lees onze disclaimer: www.accessibility.nl/algemeen/disclaimer<UrlBlockedError.aspx>
Accessibility is Member van het W3C
=========================

________________________________________
Van: Vivienne CONWAY [v.conway@ecu.edu.au]
Verzonden: donderdag 11 oktober 2012 5:02
Aan: Velleman, Eric; public-wai-evaltf@w3.org
Onderwerp: RE: Input for survey about random sampling

HI all
I think question 12 should reflect a percentage of pages rather than a finite number.  In one of the discussions with the WCAG group they mentioned that 25% of the total pages sampled should be randomly chosen.  Can we re-phrase this question to include percentages?


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
________________________________________
From: Velleman, Eric [evelleman@bartimeus.nl]
Sent: Thursday, 11 October 2012 5:18 AM
To: public-wai-evaltf@w3.org
Subject: RE: Input for survey about random sampling

Dear EvalTF,

Following our discussion last telco, the minutes of that meeting and some remarks sent to me, this is an update of the survey. Please send your comments to the list. Before we start making this a scientific research project, please keep in mind that this is input information for a first draft text to add a random sample.

** Concept survey questions about sampling **

Note: You could fill in the survey more than once using different choices in the first question. Please note that the Methodology is about full websites (as defined in the Public Working Draft).

#01. Please choose the situation below for which you fill in this survey.
Your evaluation activity is:
 - (checkbox) In-house website evaluation for your own organization
 - (checkbox) Website evaluation for external evaluation commissioner
 - (checkbox) A re-evaluation of a website that you have evaluated less than 1 month ago
 - (checkbox) In depth evaluation
 - (checkbox) Basic evaluation
 - (checkbox) Detailed evaluation
 - (checkbox) Other - please specify below (text)

Please fill in the questions depending on the evaluation activity you have chosen in question #01:

#02. For which purposes do you evaluate websites:
 - (checkbox) Provide in-depth analyses to guide website owners
 - (checkbox) Monitor the accessibility of a website over time
 - (checkbox) Final check before releasing or purchasing a website
 - (checkbox) Provide label or other certificate of conformance
 - (checkbox) Do large-scale evaluation of many websites
 - [checkbox] Pre-sales activities (evaluating websites in order to initiate a contract)
 - [checkbox] Evaluate websites in early/medium development stages
 - [checkbox] Deliver training about Web accessibility evaluation
 - (checkbox) Other - please specify below (text)

#03. Do you consult with the evaluation commissioner regarding the sample selection?
 -(radio) Always
 -(radio) Sometimes
 -(radio) Never
 -(radio) Depend upon whether the website is being done without their knowledge

#04. To what extent do the wishes of the evaluation commissioner influence the evaluation sample? 1= no influence (evaluator is independent in choice of evaluation sample), 5=total influence (evaluator is not independent in choice of of evaluation sample)
 -(radio) 1
 -(radio) 2
 -(radio) 3
 -(radio) 4
 -(radio) 5

#05. Do you select sample pages for evaluation based on:
 - (radio) Structured sampling only
 - (radio) Random sampling only
 - (radio) Automated sampling only
 - (radio) Both structured and random sampling
 - (radio) Both structured and random sampling supported by automated sampling

#06. Please specify any tools you use to help you select the sample, and how you use these tools for sampling. (text)

#07. If you use automated sampling only, how do you check the tool? (text)

#08. If you use structured sampling, how do you select the individual web pages? (text)

#09. If you use random sampling, how do you select the individual web pages? (text)

#10. How does re-evaluation of a website influence the sample? (text)

#11. Please specify if you sample pages differently or in different amounts for Basic Reporting, Detailed Reporting or In-Depth Analysis evaluation (text).

#12. What is the amount of pages you use in your sample when evaluating a website?
 - (radio) 1 - 5
 - (radio) 6 - 15
 - (radio) 16 - 30
 - (radio) 31 - 50
 - (radio) 51 - 100

#13. If the amount of pages in your evaluation sample is variable, what are –in most cases – the factor(s) that have the most influence on the size of the sample:
 - [checkbox] Size of the website
 - [checkbox] Number of technologies on the website
 - [checkbox] Amount of money available for the evaluation
 - [checkbox] Number of templates
 - [checkbox] Other, please specify below (text)

#14. How does the size of the website influence your selection? (text)

#15. How do you adjust your sampling approach for websites that are heavily template-driven? (text)

#16. If you use automated testing tool(s), does that change your sampling methodology?
 - (radio) Yes
 - (radio) No
 - Please explain (text)

#17. How do you adjust your sampling approach for web applications? (text)

#18. How do you sample mobile applications? (text)

#19. Do you look out for widgets, snippets, and other web page components to avoid re-evaluating them every time they appear on a website?
 - (radio) Yes
 - (radio) No
If Yes, how do you include them and the pages that they are integrated into the sample? (text)

#20. How do you identify and sample the functionality (processes) provided on a website? (text)

#21. Do you select further sample pages based upon results from an initial selection (eg. to balance web page types or success criteria)? (text)

#22. Please feel free to provide further thoughts or comments about how you approach sampling. (text)

Kindest regards,

Eric

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B





________________________________
This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B

******************* IMPORTANT ! *******************************
The content of this email was found to
contain potentially hostile or malicious content.
For your protection, eSafe’s Content Security Server has
modified this email and removed the dangerous content.
**********************************************************************

  Found Spoofed Phishing URL, Overall #6 Attributes Removed: href