Re: Randomly choosing pages

For me – the random sample is just another check that I do after I have checked everything useful such as landing pages, processes etc (Steps 3a, 3b, 3c and 3d in our section 3.3 of the methodology). In practice I use the random test just to check that I have not missed something. 

My structured testing (e.g. steps 3a-d) tells me if the site performs its function/s in an accessible way so I am only looking to see if I can find odd items such as bits left over from a previous version of the site or perhaps something that has been added by way of experiment. Most times my random sample just confirms my conclusions from the structured testing, occasionally it throws up a better example of a barrier that I can use in my report. Sometimes (rarely) it throws up a completely new section such as a blog or video gallery that the owner had forgotten about.

Thus I do not see the random sample as critical to the evaluation – but I do see it as a “nice to have” extra that adds value and confidence. I would support including random sampling as step 3e.

As for the method – I simple run a robot to make a list of the URLs. If that works I use a random table to find the pages to look at.

Regards
Richard

From: Kathy Wahlbin 
Sent: Friday, September 14, 2012 1:21 PM
To: Alistair Garrison ; Eval TF 
Subject: RE: Randomly choosing pages

Hi Alistair –

 

I agree with you that looking at Google search results could be some help in determining what should be reviewed.  I think this should not be the only thing we rely on however because of the limitations I noted.

 

All of this is very complex and I think will need much discussion.  The point of the random sample is to determine if the site is accessible and if the level of accessibility found on the pages reviewed are also accessible.  If you limit this to pages that are most frequently used, you will not be able to gauge the level of accessibility on the site.  I do agree that these pages are very important and should be fully accessible.  These pages should be included in the random sample but it should not be limited to these types of pages.

 

Best,

 

Kathy

T (978) 443-0798  F (978) 560-1251  C (978) 760-0682
E kathyw@ia11y.com  

www.InteractiveAccessibility.com

 

NOTICE: This communication may contain privileged or other confidential information. If you are not the intended recipient, please reply to the sender indicating that fact and delete the copy you received. Thank you.

 

From: Alistair Garrison [mailto:alistair.j.garrison@gmail.com] 
Sent: Friday, September 14, 2012 7:56 AM
To: Kathy Wahlbin; Eval TF
Subject: Re: Randomly choosing pages

 

Hi, 

 

Random sounds very complex - and quite possibly pointless. Let's say you spend an age using tools to collect random pages - if the pages surrounding the random pages are inaccessible these pages will never actually be reached by real life users.  

 

A more useful approach (which was what I was trying with the Google entry pages) was to suggest we look at the most common direct entry pages to the site, in addition to the others in our sample - whatever it does not do the google list does represent the most common pages returned in searches - and as such the most common direct entry pages.  And, direct entry pages are very worthwhile checking for accessibility.

 

Just my thoughts.

 

All the best 

 

Alistair

 

On 14 Sep 2012, at 13:38, Kathy Wahlbin wrote:





Hi -

 

I like that approach Kerstin.  Identifying the clusters within the site would be a good starting place but we would still need to decide how to pick the pages within that cluster.

 

I would not use Google search for a couple of reasons:

1.      Will not work for applications.  An application is usually behind a login and so it is blocked by Google search and will not be indexed.

2.      Pages can be block Google search bot and these pages will not be indexed.  So there could be some pages or sections of sites would be missed.

3.      The pages that show up in Google are just pages that have been optimized well for SEO and do not necessarily reflect the traffic or popularity of the page. 

 

Here are some things that I have been thinking about regarding random sampling:

 

Any approach is going to use multiple tools to help identify pages, clusters, most frequently used pages, size of sample etc. 

 

To determine which pages to review in the random sample, we can use these type of tools to determine the “pages” and frequency of use.

1. Using a website analytics program to identify the most frequently used pages

2. Use website crawler to identify all the pages in the site (this will work for websites but not applications).  The tools that create the sitemap.xml may be a good tools for this.

 

With applications, we will need to thing about the core application or platform vs. data that goes into it. We will also need to think about how to determine the size of an application that could have pages with dynamic content or pages that change with different data.

 

Another thing we need to think about is how many pages to include in a random sample.  This should be determined based on the number of pages in the website / application.  Here is a link to a calculator for statistical sampling for surveys.  This kind of tool could be used (http://www.macorr.com/sample-size-calculator.htm) to calculate the number of pages that should be reviewed for websites/applications.  For example, if we chose a 90% confidence level and a 5% confidence interval which means that you have a  85% - 95% confidence that the other pages would conform to the guidelines (if the random sample conforms).  This would mean that in a website or application we would have to have a “random” sample of the following size:

 

      Number of Pages in the Website
     Sample Size
     
      5
     5
     
      10
     10
     
      25
     23
     
      50
     42
     
      100
     73
     
      125
     86
     
      150
     97
     
      200
     116
     
      250
     131
     
      350
     153
     
      500
     176
     
      750
     200
     
      1000
     214
     

Of course we would need to decide what confidence level and confidence interval is right.  For those who are not familiar with statistical sampling, here are a few definitions (with my interpretation to websites):

 

Confidence Level

This number represents how sure you can be of the results. It is expressed as a percentage and represents how often the webpages would be expected to conform to the accessibility guidelines. A 90% confidence level means that you can be 90% certain that the website conforms to the accessibility guidelines.

Confidence Interval

This is the plus-or-minus figure for the confidence level. For example, a 5% confidence interval and a 90% confidence level means that you can have a 85% (90-5) to 95% (90+5) confidence that the page would conform to the guidelines.

Population

This is the total number of HTML pages or versions of the pages included in the website or application being reviewed.

 

One last thing that I feel we need to consider here is how automated accessibility checkers could play a role in determining the level of accessibility in addition to the “random” sample.

 

We have  a lot to think about regarding random sampling and a lot of discussion. 

 

Best,

 

Kathy

T (978) 443-0798  F (978) 560-1251  C (978) 760-0682

E kathyw@ia11y.com  

www.InteractiveAccessibility.com

 

NOTICE: This communication may contain privileged or other confidential information. If you are not the intended recipient, please reply to the sender indicating that fact and delete the copy you received. Thank you.

 

 

-----Original Message-----
From: Kerstin Probiesch [mailto:k.probiesch@gmail.com] 
Sent: Friday, September 14, 2012 3:48 AM
To: 'Vivienne CONWAY'
Cc: 'Eval TF'
Subject: AW: Randomly choosing pages

 

Hi Vivienne, all,

 

I think the importance of random sampling is much clearer when we don't think in "pages". Especially when we think about very huge sections (for example subdomains) and different groups of editors or different editors a proper random sample can make sure that an evaluator tests not only the accessibility of content which was edited by the same editor/s.

 

There are different possibilities of random sampling. One of course is having all pages in the same sample space and choose randomly (with a

script) X pages and check the edited content of those X pages.

 

Another sampling procedure is cluster sampling. Cluster sampling could be the following:

 

1. Identify the clusters of a website (for example: subdomains, sections according to the main points in a navigation bar) 2. Choose X pages out of every cluster and make sure that all relevant SCs are checked on the randomly selected pages of every identified cluster. This could for example mean: Check 1.1.1, 1.3.1, ... in the content of X pages of every cluster.

 

I think that it is not necessary to check all on every page of the random sample. The evaluator has already checked for example the navigation bars or other global elements like the footer. So the random sample is more for the edited content and not for the page in whole.

 

What random sampling should avoid is oversampling and undersampling in different contexts of the evaluation process. oversampling as well as undersampling are relevant sampling errors. Thinking about a script which chooses 5 pages (just for saying a number) out of the sample space (the whole website) every content of every page has the same probability to be selected. But: in the same time this procedure would be undersampling if a website has for example 10 subdomains.

 

Just some ideas about random sampling

 

Cheers

 

 

Kerstin

 

 

> -----Ursprüngliche Nachricht-----

> Von: Vivienne CONWAY [mailto:v.conway@ecu.edu.au]

> Gesendet: Freitag, 14. September 2012 08:37

> An: Detlev Fischer

> Cc: Eval TF

> Betreff: RE: Randomly choosing pages

>

> Hi Detlev and TF

>

> I'm with you on this one.  I'm just about to start a large audit and

> thought I'd put this into practice, but for the life of me I can't see

> an easy way to find 6 or 7 truly random pages.  I've suggested to one

> of the automated tool companies that they build this feature into

> their crawling options so that the tool would randomly choose a number

> of pages stipulated by the evaluator, and then that evaluator could

> also manually assess those same pages.  Until then however, I have no

> idea how it would be truly 'random'.  I don't think 'random' is

> supposed to mean me just saying 'I think this one will do'.  We're

> already selectively targeting pages that we've identified as critical

> to the operation of the website, use cases, complete paths etc.  I

> have no idea what we will be able to do with this requirement.

>

>

> Regards

>

> Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs) PhD Candidate &

> Sessional Lecturer, Edith Cowan University, Perth, W.A.

> Director, Web Key IT Pty Ltd.

> v.conway@ecu.edu.au

> v.conway@webkeyit.com

> Mob: 0415 383 673

>

> This email is confidential and intended only for the use of the

> individual or entity named above. If you are not the intended

> recipient, you are notified that any dissemination, distribution or

> copying of this email is strictly prohibited. If you have received

> this email in error, please notify me immediately by return email or

> telephone and destroy the original message.

> ________________________________________

> From: Detlev Fischer [detlev.fischer@testkreis.de]

> Sent: Friday, 14 September 2012 2:33 PM

> To: Vivienne CONWAY

> Cc: Eval TF

> Subject: Re: Randomly choosing pages

>

> Hi Vivienne,

>

> I remember we have discussed this already at length without ever

> coming to a sound conclusion. I suggest a practical perspective: If it

> is to be mandatory that a part of the sample is found in a true random

> process, this imposes quite a hard requirement on the evaluator:

>

> 1) He/she has to judiciously apply some crawling tool to ensure that

> the applicable scope is fully crawled and all pages are included in

> the set (excluding those that are chosen by other means) - and the

> scope pf evaluation may include not just one simple hierarchical tree

> but several sub-domains, generated pages that even don't exist without

> user input, etc, so it is rarely an easy task, and quite hard for

> complex sites;

>

> 2) Then he/she has to apply a random procedure to the complete set of

> pages/ states within the scope by applying some random choice tool

>

> I remember some of these tools were said to exist and might be put to

> practice, but the overhead of work seems inordinate for the added

> benefit of having a few truly random pages included. And all this

> hinges on the ability and means to verify that a truly random

> procedure has indeed be applied. Who is going to check this, from the

> outside? To enagle independent verification would mean that the

> crawing and selection stages and tools will have to be documented for

> the process to be potentially 'replicable' (with different results of

> course, otherwise it would not be truly random). And if (more than

> likely) *now one* will be willing and able to ever check, we are just

> left to *believe* that the 'random pages' were indeed chosen by true

> random sampling. The concencious ones will go to a lot of trouble for

> something unverifiable, the less conscientious ones will just take an

> informal 'random pick' and claim the pages were chosen 'at random'

> (which might even be true in the colloquial sense of the word).

>

> I still don"t see the added benefit of making additional random

> sampling a mandatory (methodology) requirement...

>

> Just my 2 cents, as they say - Detlev

>

>

>

> On 14 Sep 2012, at 05:11, Vivienne CONWAY wrote:

>

> > Hi all

> >

> > As we're giving some thought to the inclusion of randomly selected

> > pages for part of the sample, I'm wondering how people propose the

> > evaluator would generate the randomly chosen pages.

> >

> > Any thoughts?

> >

> >

> > Regards

> >

> > Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs) PhD Candidate &

> > Sessional Lecturer, Edith Cowan University, Perth, W.A.

> > Director, Web Key IT Pty Ltd.

> > v.conway@ecu.edu.au

> > v.conway@webkeyit.com

> > Mob: 0415 383 673

> >

> > This email is confidential and intended only for the use of the

> > individual or entity named above. If you are not the intended

> > recipient, you are notified that any dissemination, distribution or

> > copying of this email is strictly prohibited. If you have received

> > this email in error, please notify me immediately by return email or

> > telephone and destroy the original message.

> > ________________________________________

> > From: Shadi Abou-Zahra [shadi@w3.org]

> > Sent: Friday, 14 September 2012 5:07 AM

> > To: Eval TF

> > Subject: Minutes for Teleconference on 13 September 2012

> >

> > Eval TF,

> >

> > Please find the minutes for the teleconference on 13 September 2012:

> >  - <http://www.w3.org/2012/09/13-eval-minutes>

> >

> > Next meeting: Thursday 20 September 2012.

> >

> >

> > Regards,

> >   Shadi

> >

> > --

> > Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead,

> > W3C/WAI International Program Office Evaluation and Repair Tools

> > Working Group (ERT WG) Research and Development Working Group (RDWG)

> >

> > This e-mail is confidential. If you are not the intended recipient

> > you must not disclose or use the information contained within. If

> > you have received it in error please return it to the sender via

> > reply e-mail and delete any record of it from your system. The

> > information contained within is not the opinion of Edith Cowan

> > University in general and the University accepts no liability for

> > the accuracy of the information provided.

> >

> > CRICOS IPC 00279B

> >

>

> --

> Detlev Fischer

> testkreis - das Accessibility-Team von feld.wald.wiese c/o

> feld.wald.wiese Thedestraße 2

> 22767 Hamburg

>

> Tel   +49 (0)40 439 10 68-3

> Mobil +49 (0)1577 170 73 84

> Fax   +49 (0)40 439 10 68-5

>

> http://www.testkreis.de

> Beratung, Tests und Schulungen für barrierefreie Websites

>

> This e-mail is confidential. If you are not the intended recipient you

> must not disclose or use the information contained within. If you have

> received it in error please return it to the sender via reply e-mail

> and delete any record of it from your system. The information

> contained within is not the opinion of Edith Cowan University in

> general and the University accepts no liability for the accuracy of

> the information provided.

>

> CRICOS IPC 00279B