Re: Feedback on WCAG-EM January 2014 WD

Hi Eric,

I hope you find them useful. Yes, feel free to contact me if you have additional questions.

Regarding the test-run, I will take this question to the Working Group. It will depend on available time and resources.

Best regards,

Jason Kiss
Senior Advisor - Digital Engagement
Government Information Services
Information and Knowledge Services
The Department of Internal Affairs - Te Tari Taiwhenua
Direct Dial: +64 (04) 462 0332
Extension: 3732
Email: jason.kiss@dia.govt.nz
http://www.dia.govt.nz
http://webtoolkit.govt.nz/


On 28/02/2014, at 10:32 pm, Velleman, Eric <evelleman@bartimeus.nl> wrote:

> Hello Jason,
> 
> Many thanks to you and your colleagues for the valuable feedback on WCAG-EM. In the coming period we will discuss them in the Evaluation Taskforce. The outcome of the discussions will be visible in a Disposition of Comments where we will address all comments. Is it ok if we contact you in case we have additional questions about the comments you made?
> 
> We also plan to do a test-run of WCAG-EM soon. For this we ask to evaluate a website provided by Eval Taskforce using WCAG-EM. Would you or your colleagues be interested to participate in such a test-run?
> 
> Kindest regards,
> 
> Eric Velleman
> 
> ________________________________
> Van: Jason Kiss [Jason.Kiss@dia.govt.nz]
> Verzonden: vrijdag 28 februari 2014 0:29
> Aan: 'public-wai-evaltf@w3.org'
> Onderwerp: Feedback on WCAG-EM January 2014 WD
> 
> The following comments, some editorial and some more substantive, have been provided by individual members of the New Zealand Government Web Standards Working Group and do not necessarily represent the position of the New Zealand Government Web Standards Working Group as whole, the Department of Internal Affairs, or the New Zealand Goverment.
> ---
> Scope of Applicability
> Principle of Website Enclosure
> The Principle of Website Enclosure is not referenced anywhere else in the methodology document and seems redundant given the remainder of context in this section. Document would flow better by dropping the Principle of Website Enclosure heading and the paragraph immediately below it and using the remainder of the content (including the diagram) in the section as an appendage to the section preceding it. The remaining content illustrates the application of the description in the previous section anyway.
> Particular Evaluation Contexts
> Re-Running Website Evaluation
> The phrase "Additional web pages" in the second list item suggests that extra web pages need to be added to the previous sample, thus increasing the overall number of pages in the sample to be assessed. If the evaluation is run multiple times, the sample size could potentially get unwieldy or impractically large. If this is not what is intended, it's suggested that "Additional web pages" be replaced with "Different web pages" in alignment with the phrasing used in the third bullet point. It might also be useful to note explicitly that the pages in the sample should be updated/changed without the overall sample size needing to increase.
> Evaluation Procedure
> Step 2: Explore the Target Website
> Step 2c: Identify the variety of Web Page Types
> In the bulleted list items, the phrasing occasionally gets awkward, e.g., "Content with varying types of content", "Content that are", "Content with dynamic content". One solution may be to change the word "Content" to "Pages", to read as follows:
> Content to look for to identify different types of web page and web page states include:
> ·         Pages with varying styles, layout, structure, ...
> ·         Pages with varying types of content such as ...
> ·         Pages ...
> A second proposal is to change the text as follows:
> Things to look for to identify different types of web page and web page states include:
> ·         Content with varying styles, layout, structure, navigation, interaction, and visual design;
> ·         The use of varying types of content such as forms, tables, lists, headings, multimedia, and scripting;
> ·         The use of varying functional components such as date picker, lightbox, slider, and others;
> ·         Content using varying technologies such as HTML, CSS, JavaScript, WAI-ARIA, PDF, etc.;
> ·         Different website areas (home page, web shop, departments, etc.) including any applications;
> ·         Content created using different templates (if this is known to the evaluator);
> ·         Content authored by different people (if this is known to the evaluator);
> ·         Content created using different coding styles;
> ·         Content that changes appearance, behavior, and content depending on the user, device, browser, context, and settings;
> ·         The use of dynamic content, error messages, dialog-boxes, pop-up windows, and other interaction.
> Step 2.e Identify Other Relevant Web Pages
> The Methodology Requirement is labelled 3.c ... shouldn't it be 2.e?
> Step 3: Select a Representative Sample
> The introductory paragraph states "The purpose ... the accessibility performance of the website with reasonable confidence." The word reasonable is highly subjective. It should possibly refer to an acceptable level of confidence, with the definition of acceptable set by the evaluation commissioner. The methodology might benefit from a discourse on levels of confidence which balances out statistical levels of confidence (which are good for random samples) with the practical reality that sites contain many repeated patterns and practices.
> Step 3.e: Include a Randomly Selected Sample
> It seems clear from the methodology that the purpose of the randomly selected sample is to buttress and help ensure the representativeness of the structured sample, but this is not clearly expressed in Step 3.e, which confusingly suggests that the randomly selected sample helps to verify that the structured sample reflects the "accessibility performance of the site". But the "accessibility performance of the site" can't be assessed without having first performed the evaluation of the pages included in the total sample. Suggest rephrasing first sentence as follows:
> "A randomly selected sample of web pages and web page states acts as an indicator to verify that the structured sample selected through the previous steps is sufficiently representative of the content provided on the website."
> This would make it clearer and more consistent with the language used in Step 4.e.
> There may also be better value for money to be gained from skewing such a sample towards most visited pages. If the randomly selected sample is to be 10% of the structured sample, something like 5% randomly drawn from the top 20%(?) of pages accessed, the other 5% selected from the remaining 80%. A recommendation to this effect could be useful.
> A typo:
> Confidence in the overall evaluation outcome increases when the evaluation results from both selection [approaches] correlate.
> Step 4: Audit the Selected Sample
> Step 4.d: Check for Non-Interference
> This section would benefit from a brief description of the requirements for non-interference.
> Step 5: Record the Evaluation Findings
> Step 5.a: Document the Outcomes of Each Step
> The Note: section allows for repeated failures of a Success Criterion to be reported once with one example. For the purposes of re-evaluation in particular, but also to identify persistent weaknesses in knowledge or process, it would be useful to recommend a preference that each repeated instance of a failure be reported back to the evaluation commissioner.
> Step 5.d: Provide an Aggregated Score
> This section should be heavily alarmed with warnings that reinforce the message that accessibility cannot be given a score, and that the real (only?) purpose of such a metric is to provide a unit-less coefficient that can be used to indicate improvement over time. There is a significant risk that 'scores' like this may get used further up management structures as hard metrics, which they are not.
> Aggregated score per website
> Suggest changing "not present in" to "not applicable to" in the following:
> 4. Mark the remaining WCAG 2.0 Success Criteria as [not applicable to] any web page or web page state in the sample selected in Step 3: Select a Representative Sample.
> Typo:
> This particular score [is] sensitive towards failures...
> Aggregated score per web page
> Suggest changing "not present" to "not applicable" in the following:
> ·         Mark the WCAG 2.0 Success Criteria that are [not applicable]
> Typo:
> For example, a single failure will yield a high score even if that one [failure] is an inaccessible CAPTCHA that prevents access to the entire website.
> 
> ---
> 
> 
> 
> Regards,
> 
> Jason Kiss
> Senior Advisor - Digital Engagement
> Government Information Services
> Information and Knowledge Services
> The Department of Internal Affairs - Te Tari Taiwhenua
> Direct Dial: +64 4 462 0332
> Extension: 3732
> Email: jason.kiss@dia.govt.nz<mailto:jason.kiss@dia.govt.nz>
> http://www.dia.govt.nz<http://www.dia.govt.nz/>
> https://webtoolkit.govt.nz/<http://webstandards.govt.nz/>
> 
> 
> 
>

Received on Saturday, 1 March 2014 03:45:06 UTC