W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > February 2012

Re: Evaluation scheme with three options - proposal

From: Shadi Abou-Zahra <shadi@w3.org>
Date: Mon, 20 Feb 2012 16:43:23 +0100
Message-ID: <4F426A1B.2090402@w3.org>
To: public-wai-evaltf@w3.org
Small addition:

On 20.2.2012 16:28, Shadi Abou-Zahra wrote:
> Hi Kerstin, All,
>
> I'm not too sure what the difference between options #1 and #2 would be
> in practice, as I hope that evaluators will simply link to Techniques
> rather than to attempt to explain the issues themselves.
>
>
> Here is an example of what a report of option #1 could look like:
> - <http://www.w3.org/WAI/demos/bad/before/reports/home.html>

Here is a positive example too: ;)
  - <http://www.w3.org/WAI/demos/bad/after/reports/home.html>


Regards,
   Shadi


> Note: this is a report for a single page but it could still be a basis
> for reports of option #1 for entire websites; it just has a pass/fail
> for each Success Criterion and some Techniques to justify these claims.
>
>
> For option #2 we could introduce a scoring function in addition to the
> pass/fail result. This would require the evaluators to fully evaluate
> every page in the selected sample and count the frequencies of errors to
> calculate a score. It could help compare websites and motivate the
> developers (at least those who are close to full compliance).
>
>
> Finally, option #3 would be more in-depth reports with examples of the
> errors and explanations of ways to repair the errors. These are, as
> Kerstin says, developed by consultants (as opposed to pure evaluators)
> for developers who are new to accessibility.
>
> We attempted to provide such an example report in the initial version of
> the Before and After Demo (BAD) but it is really lots of work:
> - <http://www.w3.org/WAI/EO/2005/Demo/report/>
>
>
> Regards,
> Shadi
>
>
> On 19.2.2012 20:36, Elle wrote:
>> Kerstin:
>>
>> I like these three options. I am interested, however, in how many clients
>> that typically ask for something as abbreviated as Option 1. For those in
>> this group, do you experience situations with a lot of clients who don't
>> want more than the pass/fail report?
>>
>>
>>
>> Regards,
>> Elle
>>
>>
>>
>>
>> On Sun, Feb 19, 2012 at 4:36 AM, Kerstin Probiesch<
>> k.probiesch@googlemail.com> wrote:
>>
>>> Hi all,
>>>
>>> in our last teleconference we discussed a evaluation scheme with three
>>> options based upon 100% Conformance. I appreciate these proposals and
>>> see
>>> them as chance to integrate or point to the three documents of WCAG2:
>>> Guidelines and SCs, Understanding and How to meet.
>>>
>>> One proposal for handling the documents in an evaluation scheme,
>>> based upon
>>> the normative guidelines and SCs as core:
>>>
>>> =====
>>> Option 1: WCAG 2.0 – Core Test ("light version" or whatever the wording
>>> later will be)
>>>
>>> # Guideline X (Heading)
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) - – if regional: a
>>> list of
>>> pages where the problem exists
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) - – if regional: a
>>> list of
>>> pages where the problem exists
>>>
>>> (...)
>>>
>>> =====
>>>
>>> Use cases for Option1:
>>>
>>> - experienced developers and clients who know WCAG2 and need just the
>>> results,
>>> - comparative evaluations (20 hotel websites, city websites…)
>>> - or for example just with the SCs of level a and a smaller scope as
>>> pre-test to decide together with the client what the best next steps
>>> might
>>> be (evaluation, consulting, probably workshops for editors)
>>>
>>> =====
>>>
>>> Option 2: WCAG 2.0 – Core incl. understanding (name?)
>>>
>>> # Guideline X (Heading)
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) – if regional: a list of
>>> pages where the problem exists
>>>
>>> Problem (Subheading): Description of existing problems and barriers for
>>> users (here know how out of the understanding document could be part
>>> of the
>>> description).
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) – if regional: a list of
>>> pages where the problem exists
>>>
>>> Problem (Subheading): Description of existing problems and barriers for
>>> users (here know how out of the understanding document could be part
>>> of the
>>> description).
>>>
>>> (...)
>>>
>>> ======
>>>
>>> Use cases:
>>>
>>> - comparative evaluations (depending on the specific time and costs)
>>>
>>> - if a client just want descriptions
>>>
>>> - regular tests like "evaluation of the week"
>>>
>>> =====
>>>
>>> Option 3: WCAG 2.0 – Core, understanding, how to meet (name?)
>>>
>>> # Guideline X (Heading)
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) – if regional: a list of
>>> pages where the problem exists
>>>
>>> Problem (Subheading): description/explanation of existing problems and
>>> barriers for users (here know how out of the Understanding Document
>>> could
>>> be
>>> part of the description).
>>>
>>> Action (Subheading): Description of techniques for meeting the SC
>>> (could be
>>> techniques which are already in the techniques document or new
>>> techniques
>>> which are not in the document, but with which the SC can be met).
>>> Here even
>>> usability aspects can play a role, like: you can do a, b, c or d – I/we
>>> propose/recommend c.
>>>
>>> ## Checkpoint: SC XX (Subheading)
>>>
>>> Result: pass/fail
>>>
>>> Character: global/regional (or another wording) – if regional: a list of
>>> pages where the problem exists
>>>
>>> Problem (Subheading): description/explanation of existing problems and
>>> barriers for users (here know how out of the Understanding Document
>>> could
>>> be
>>> part of the description).
>>>
>>> Action (Subheading): Description of techniques for meeting the SC
>>> (could be
>>> techniques which are already in the techniques document or new
>>> techniques
>>> which are not in the document, but with which the SC can be met).
>>> Here even
>>> usability aspects can play a role, like: you can do a, b, c or d – I/we
>>> propose/recommend c.
>>>
>>> (...)
>>>
>>> ======
>>>
>>> Use cases:
>>>
>>> - test incl. consulting
>>>
>>> - for clients who are not very familiar with accessibility and WCAG2
>>>
>>> ============
>>>
>>> For a seal/badge or any formal confirmation Option 1 is the minimum.
>>>
>>> A report might also / should? also have intro parts like:
>>>
>>> - Short description of the Option 1, 2 or 3
>>>
>>> - Something like a disclaimer ("results might not be complete,
>>> therefore it
>>> is important to go through the page, view all similar elements and solve
>>> the
>>> corresponding problems)
>>>
>>> - Glossary (for specific terms we used in our methodology -like
>>> regional/global – if we decide to use them)
>>>
>>> - Documentation of the used OS, Browsers and Versions, probably used
>>> assistive technologies incl. versions
>>>
>>> - Tested Conformance Level (A, AA, AA)
>>>
>>> - Results
>>>
>>> - Summary, probably written as an overall impression - we discussed
>>> in this
>>> list the 'motivation factor'. I think the aim of an evaluation is not to
>>> motivate. Nevertheless, writing a nice overall impression in a
>>> report, may
>>> have this function. Ok, except when there is nothing nice to say.
>>>
>>> This scheme could probably also be used for processes, pdf, flash and
>>> so on
>>> and I think it would be flexible enough (time, costs, ...) and in the
>>> same
>>> time valid against the Conformance Requirements, because the core
>>> (evaluation itself) is the same in every option.
>>>
>>> Important, as I see it, is that the evaluator has the three different
>>> aspects in mind and in the report, which I believe shouldn't be mixed:
>>> evaluation (Core, testing SCs), explanation (description of the
>>> problem/violation, understanding) and consulting (how to meet,
>>> usability,..)
>>>
>>>
>>> The evaluator could document the "progress toward meeting success
>>> criteria
>>> from all levels beyond the achieved level of conformance": If for
>>> example
>>> the evaluation is for Level A with Option 3 the SCs of AA could also be
>>> checked (pass/fail) without any further description or with further
>>> description, depending on the contract.
>>>
>>> Advantage: every evaluator or testing organization uses the
>>> methodology and
>>> a standardized 'template' for the core and the evaluation itself. The
>>> descriptions of existing barriers (explanatory part/understanding in
>>> Option
>>> 2 and 3) and the consulting part (How to meet, in Option 3) would be the
>>> specific added value for the clients/the evaluator/the testing
>>> organization.
>>>
>>>
>>> Thoughts?
>>>
>>> Best
>>>
>>> --Kerstin
>>>
>>>
>>> -------------------------------------
>>> Kerstin Probiesch - Freie Beraterin
>>> Barrierefreiheit, Social Media, Webkompetenz
>>> Kantstraße 10/19 | 35039 Marburg
>>> Tel.: 06421 167002
>>> E-Mail: mail@barrierefreie-informationskultur.de
>>> Web: http://www.barrierefreie-informationskultur.de
>>>
>>> XING: http://www.xing.com/profile/Kerstin_Probiesch
>>> Twitter: http://twitter.com/kprobiesch
>>> ------------------------------------
>>>
>>>
>>>
>>
>>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)
Received on Monday, 20 February 2012 15:43:52 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:13 GMT