Testing website #1 responses

Please find below the summary of results for testing website #1

In total seven people participated in the questionnaire, four reported some of their results and three simply supplied comments.


The first three questions asked the testers to complete the first three steps of the methodology and report their results.

Step 1: Define the Evaluation Scope
Seven testers responded. 
Four testers reported results for this step. All clearly identified the base url, goal and conformance target. One tester included some external pages for donations and privacy statements within the scope, another tester specifically excluded these sections.

The three testers who did not report results, and two of the four who did, expressed concern that the guidance was not as clear as it could be.
 Not clear what to document *1
 Techniques and failures needs clarifying *3
 Meaning of “unambiguous”? *2

Step 2: Explore the Target Website
Only one tester completed all four steps. A second testers identified common web pages (2a) and common functionality (2b) with similar results. 
Two testers only identified the technologies relied upon (2d)
One tester prepared a list of pages but did not enter the results.

Step 3: Select a Representative Sample
Three testers completed this question with their results. 
 One listed a selection of 26 page urls (including 12 pages from the forum application and 1 paypal page) , 
 one listed a selection of 11 pages with descriptions and urls 
 one selected the whole site (25 pages) plus a selection of forum pages.


The fourth question asked the tester to describe how the sample of pages was gathered
4    Please describe how you gathered the sample
Only three testers replied 
     One tester made an early decision based on step 2 to include the whole website. 
 One tester did a Google search to check for any missed pages before deciding to include the whole site. 
 A third tester classified pages by category, but did not include their results so it is not clear if they covered the site or how they selected the sample.


The remaining thirteen questions asked the testers to report on the usability of the methodology. 
Question 5, 8 and 13 asked basically the same question (is the guidance clear) so I have combined their answers
5   Do you find the guidance clear, does it cover the items you feel it should?
8   Is the guidance clear and unambiguous?
13    Are steps 1, 2 and 3 clear, and easy to follow?
 Guidance is clear  *1
 Guidance is reasonably clear *1
 Guidance is confusing for step 3 (methodology)*1
 Need something in step 2 defining third party tools *1


6    Does the guidance cover websites as defined (including web applications etc.)?
Yes * 2
Additional guidance required *1


7 Does the guidance cover very large and very small websites?
Possibly *2

8    (see question 5 above)

9    Do multiple different evaluators end up with comparable results when using WCAG-EM?
Not answerable yet

10 Do you find any guidance or information missing from the guidance that is important for understanding?
Need more guidance on scope (step 1) *1

11 Do you find any guidance or information missing that is important for the practical use of WCAG-EM?
Would like more info *2

12 Do you find anything in the guidance that in any way contradicts WCAG 2.0?
No *2

13    (See above – question 5 )
14    Is there anything you feel should be added to one or more of the reports? Or removed?
Add the rationale for the sample *1

15 Is the format of the report clear and easy to follow?
Yes *3

16 Were there parts of the evaluation (following the guidance) that were more difficult to accomplish?
Setting scope was difficult *1

17 Do you feel they were able to collect a representative sample of the pages of the website (or paths through a web application)?
Yes *2

18   Is the size of the sample you developed in line with the "typical" sample size you would have otherwise used ?
Yes *1
Larger *2

----------------------------------------------------------------------------------



Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com