Revised questionnaire

As requested in the conference I am posting my thoughts on the actual questionnaire that we are using. I would truly value your opinions so that, perhaps, we can make improvements to the questionnaire and the level of responses.

Having analysed the responses I have some comments about the actual questionnaire that we used. I feel that the poor response may be due to the questionnaire being longer than needed, not as clear as could be and containing some duplication (5,8 & 13) or questions that could not be answered at this time (7, 9 & 15) .

May I suggest that something like the following would be easier for testers and for analysis. If we ask testers to perform each step in turn, report the result and then comment on the step before moving to the next step we keep information together. The remaining (general) questions should only cover the most relevant issues with regard to this stage.

We might consider the following as a model for a redesign of the current questionnaire for steps 1,2 and 3.

------------------------------------------------------
1. Introduction
Please remember that we are testing the methodology, not WCAG or your personal ability to evaluate websites. As a member of the Eval task-force you fit into our profile by having the required expertise (http://www.w3.org/TR/WCAG-EM/#expertise) so please feel free to comment as much as possible.

Please follow the text in the Methodology as close as possible.

2. Step 1: Define the Evaluation Scope
Please complete step 1 of WCAG-EM and post the results below . For guidance on what to report see: Step 5
The goal of the evaluation is a XXXXX

3. Evaluation of step 1:
a) Please comment on how easy you found it to follow the guidelines for step 1 
b) Do you think that the guidelines for step 1 should be improved? If so please explain how this might be done


4. Step 2: Explore the Target Website
Please post the results of Step 2 of WCAG-EM. For guidance on what to report see: Step 5

5. Evaluation of step 2
a) Please comment on how easy you found it to follow the guidelines for step 2
b) Do you think that the guidelines for step 2 should be improved? If so please explain how this might be done

6  Step 3: Select a Representative Sample
Please post the results of Step 3 of WCAG-EM. For guidance on what to report see: Step 5

7 Evaluation of step 3
a) Please comment on how easy you found it to follow the guidelines for step 3
b) If you used a “random sample” please give a short description of how you did this (e.g “selected every tenth page from the site map”)
c) Do you feel they were able to collect a representative sample of the pages of the website (or paths through a web application)? Please say if you are “very confident”, “fairly confident” or “not confident” that your sample is representative.
d) Is the size of the sample you developed in line with the "typical" sample size you would have otherwise used ?
e) Do you think that the guidelines for step 3 should be improved? If so please explain how this might be done.

8) Evaluation of guidance given by step 5
a) Please comment on how clear the guidance in step 5 was.
b) Did you find anything missing from the guidance about what to report given in step 5

8) Do you find any guidance or information missing from the guidance that is important for understanding?

9) Do you have any other comments with regard to using the guidance for completing steps 1,2 and 3.

-----------------------------------------------------

Kind regards
Richard

Richard Warren
Technical Manager
Website Auditing Limited (Userite)
http://www.website-accessibility.com