Comments on the Website Accessibility Conformance Evaluation Methodology

SSB BART Group ("SSB") would like to thank the W3C and the Evaluation
Methodology Task Force for the opportunity to comment on the Website
Accessibility Conformance Evaluation Methodology ("EM") version 1.0 working
draft dated March 27th, 2012.



We firmly believe that a formal methodology is important to the community
of accessibility conformance evaluators and is an important step in
professionalizing our field.    SSB uses a similar methodology for our
customers and understands the need for different detail level of audits
with required and optional criteria.



The scope of the methodology is indicated to apply to full websites.  While
we applaud this effort and agree that sites should not be cherry-picked for
accessible pages or features -- it may not always be practical for entire
websites.  Thus, the idea of applying the scope to departments or smaller
portions of a site will likely be important.  For example, a particular
purpose of the site such as the online forum or reservation section of the
site and whole pages within may be targets of the evaluation.  Similarly
allowing websites to report on conformance while still have non-accessible
third party content such as social media links/log-ins/ads is a good idea.
While in principle third party content should also conform sites should be
credited with their efforts to be conformant of what they have control over.



Regarding involving review teams and users, and evaluation tools, we
believe that functional testing in addition to normative testing of content
is very important.   While normative or unit testing is good at identifying
specific issues with a widget or missing requirement, the impact of placing
widgets together on a page with a required task such as completing a form,
finding content, or even logging in may raise accessiblity issues that were
not detected in normative testing.  At SSB we create use cases based on
core tasks within the site/application in addition to capturing a
representative set of pages ("modules").  These use cases are scripted
during the groundwork phase ("Sampling phase of this EM") and later
evaluated with three main categories of assistive technology during the
evaluation phase (screen reading, screen magnification, and speech
recognition).



The goal of "Basic conformance" as stated in the EM appears to be offer an
option to essentially self certify without having to perform real
evaluation of the site.  This level of conformance is likely to be
incorrectly applied and would likely mean that the site would in fact not
be conformant.  In cases where some national legislation references WCAG
this could allow someone to argue conformance without defensible claims.
Making an assumption on conformance, while often well intentioned, is not
safe for sites that have not previously been evaluated.  The working group
should consider allowing this level of conformance only for conformance
claims that are made after updates are made to an already conforming site.



Under the section "identify the variety of page types" it is worthwhile to
strength or include different states of pages such as error states for
forms, dynamically added content, simulated dialogs or pop-ups, or page
renderings based on different user settings and permissions including
whether an accessibility enhancement settings are enabled.



In "Step 3: Select a Representative Sample", it may be useful to choose
portions of pages that are reused such as header sections, navigation
structures, etc.  While these portions are not full web pages and full web
page conformance must general be met, capturing sections of the page can be
useful in targeting violations, limiting repetitive evaluation of the same
structure on each page, etc.  An XPath or other identifier could be used to
identify these sections within a given page.   For all the pages in which
this section or widget appears violations could then be applied as pattern
violations thus saving time during the evaluation process.



Step 4d calls for the archival of the pages and capturing of screenshots.
This step is very important and is something that SSB does during the
sample collection of the site.  In addition, it may be important to record
the steps (path) used to reach a page including any data entered, user
rights, etc.



Regarding providing conformance scores Step 5c, the current wording
indicates that a score of non-conformant is likely to occur and that score
is important.  This is indeed the case as many sites support WCAG but are
not fully conformant to a given level.  WCAG requires conformance on a per
level basis with only a few exceptions.  In reality the accessibility of
the site may be experience differently to different user groups.  For
example, a site may be accessible to users with hearing impairments via
captions, visual equivalents for sound, etc. but may not be accessible or
fully conformant to WCAG success criteria related to access by persons who
are blind.  Knowing the relative score may assist users in making decisions
to use a particular site based on their needs.  While a goal of full
conformance to all criteria is key -- conformance may be an evolving
process within an organization and may change with updates to a site.
Thus, a performance metric and a unified process for its calculation is
important to this field.



Best Regards,



Jonathan



-- 
Jonathan Avila

Chief Accessibility Officer
SSB BART Group
jon.avila@ssbbartgroup.com

703.637.8957 (o)
Follow us: Facebook <http://www.facebook.com/#!/ssbbartgroup> |
Twitter<http://twitter.com/#!/SSBBARTGroup>
 | LinkedIn <http://www.linkedin.com/company/355266?trk=tyah> |
Blog<http://www.ssbbartgroup.com/blog>
 | Newsletter <http://eepurl.com/O5DP>