RE: Not Applicable (was Re: Evaluation scheme with three options - proposal)

Hi Vivienne,

Think I put it on the agenda. So lets talk about it.
Kindest regards,

Eric



________________________________________
Van: Vivienne CONWAY [v.conway@ecu.edu.au]
Verzonden: woensdag 22 februari 2012 3:13
Aan: Michael S Elledge; Shadi Abou-Zahra
CC: public-wai-evaltf@w3.org
Onderwerp: RE: Not Applicable (was Re: Evaluation scheme with three options  -  proposal)

Hi all

I just ran across this discussion which is something that I think we should put in the EVTF methodology.  I know that I've been using n/a when it seemed the item was not present in the website e.g. videos.  However if this is the W3C consensus, I'll need to change my reporting.  Can we talk about this in our teleconference this week?


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
________________________________________
From: Michael S Elledge [elledge@msu.edu]
Sent: Wednesday, 22 February 2012 2:29 AM
To: Shadi Abou-Zahra
Cc: public-wai-evaltf@w3.org
Subject: Re: Not Applicable (was Re: Evaluation scheme with three options  - proposal)

Thanks for the explanation, Shadi. I imagine it took some discussion to
reach that consensus!  :^)

Mike

On 2/20/2012 2:30 PM, Shadi Abou-Zahra wrote:
> Hi Mike,
>
> Good question. We had a long discussion about that and also asked the
> WCAG Working Group on their position on this.
>
> According to WCAG WG, the term "Not Applicable" is not defined and is
> ambiguous. Accessibility requirements are deemed met when the content
> does not require specific accessibility features. For example, the
> requirement for captioning is deemed met if there is no video content.
>
> I will try dig out the corresponding pointers but I recall that this
> was something that was less clearly documented in the WCAG documents.
> We will probably need to clarify this point somewhere in section 5 of
> the Methodology, and possibly as WCAG WG to also clarify some of their
> materials (so that we can refer to it from our explanation).
>
> Best,
>   Shadi
>
>
> On 20.2.2012 19:26, Michael S Elledge wrote:
>> Hi Shadi--
>>
>> I noticed in the BAD example that success criteria for which there was
>> no related web content received a "Pass." I'm curious why that approach
>> was chosen rather than to identify such instances as "Not Applicable" or
>> "NA." Wouldn't using the term "NA" be both more informative and
>> accurate?
>>
>> Mike
>>
>> On 2/20/2012 10:43 AM, Shadi Abou-Zahra wrote:
>>> Small addition:
>>>
>>> On 20.2.2012 16:28, Shadi Abou-Zahra wrote:
>>>> Hi Kerstin, All,
>>>>
>>>> I'm not too sure what the difference between options #1 and #2
>>>> would be
>>>> in practice, as I hope that evaluators will simply link to Techniques
>>>> rather than to attempt to explain the issues themselves.
>>>>
>>>>
>>>> Here is an example of what a report of option #1 could look like:
>>>> - <http://www.w3.org/WAI/demos/bad/before/reports/home.html>
>>>
>>> Here is a positive example too: ;)
>>> - <http://www.w3.org/WAI/demos/bad/after/reports/home.html>
>>>
>>>
>>> Regards,
>>> Shadi
>>>
>>>
>>>> Note: this is a report for a single page but it could still be a basis
>>>> for reports of option #1 for entire websites; it just has a pass/fail
>>>> for each Success Criterion and some Techniques to justify these
>>>> claims.
>>>>
>>>>
>>>> For option #2 we could introduce a scoring function in addition to the
>>>> pass/fail result. This would require the evaluators to fully evaluate
>>>> every page in the selected sample and count the frequencies of
>>>> errors to
>>>> calculate a score. It could help compare websites and motivate the
>>>> developers (at least those who are close to full compliance).
>>>>
>>>>
>>>> Finally, option #3 would be more in-depth reports with examples of the
>>>> errors and explanations of ways to repair the errors. These are, as
>>>> Kerstin says, developed by consultants (as opposed to pure evaluators)
>>>> for developers who are new to accessibility.
>>>>
>>>> We attempted to provide such an example report in the initial
>>>> version of
>>>> the Before and After Demo (BAD) but it is really lots of work:
>>>> - <http://www.w3.org/WAI/EO/2005/Demo/report/>
>>>>
>>>>
>>>> Regards,
>>>> Shadi
>>>>
>>>>
>>>> On 19.2.2012 20:36, Elle wrote:
>>>>> Kerstin:
>>>>>
>>>>> I like these three options. I am interested, however, in how many
>>>>> clients
>>>>> that typically ask for something as abbreviated as Option 1. For
>>>>> those in
>>>>> this group, do you experience situations with a lot of clients who
>>>>> don't
>>>>> want more than the pass/fail report?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Elle
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Feb 19, 2012 at 4:36 AM, Kerstin Probiesch<
>>>>> k.probiesch@googlemail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> in our last teleconference we discussed a evaluation scheme with
>>>>>> three
>>>>>> options based upon 100% Conformance. I appreciate these proposals
>>>>>> and
>>>>>> see
>>>>>> them as chance to integrate or point to the three documents of
>>>>>> WCAG2:
>>>>>> Guidelines and SCs, Understanding and How to meet.
>>>>>>
>>>>>> One proposal for handling the documents in an evaluation scheme,
>>>>>> based upon
>>>>>> the normative guidelines and SCs as core:
>>>>>>
>>>>>> =====
>>>>>> Option 1: WCAG 2.0 – Core Test ("light version" or whatever the
>>>>>> wording
>>>>>> later will be)
>>>>>>
>>>>>> # Guideline X (Heading)
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) - – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) - – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> (...)
>>>>>>
>>>>>> =====
>>>>>>
>>>>>> Use cases for Option1:
>>>>>>
>>>>>> - experienced developers and clients who know WCAG2 and need just
>>>>>> the
>>>>>> results,
>>>>>> - comparative evaluations (20 hotel websites, city websites…)
>>>>>> - or for example just with the SCs of level a and a smaller scope as
>>>>>> pre-test to decide together with the client what the best next steps
>>>>>> might
>>>>>> be (evaluation, consulting, probably workshops for editors)
>>>>>>
>>>>>> =====
>>>>>>
>>>>>> Option 2: WCAG 2.0 – Core incl. understanding (name?)
>>>>>>
>>>>>> # Guideline X (Heading)
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> Problem (Subheading): Description of existing problems and barriers
>>>>>> for
>>>>>> users (here know how out of the understanding document could be part
>>>>>> of the
>>>>>> description).
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> Problem (Subheading): Description of existing problems and barriers
>>>>>> for
>>>>>> users (here know how out of the understanding document could be part
>>>>>> of the
>>>>>> description).
>>>>>>
>>>>>> (...)
>>>>>>
>>>>>> ======
>>>>>>
>>>>>> Use cases:
>>>>>>
>>>>>> - comparative evaluations (depending on the specific time and costs)
>>>>>>
>>>>>> - if a client just want descriptions
>>>>>>
>>>>>> - regular tests like "evaluation of the week"
>>>>>>
>>>>>> =====
>>>>>>
>>>>>> Option 3: WCAG 2.0 – Core, understanding, how to meet (name?)
>>>>>>
>>>>>> # Guideline X (Heading)
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> Problem (Subheading): description/explanation of existing
>>>>>> problems and
>>>>>> barriers for users (here know how out of the Understanding Document
>>>>>> could
>>>>>> be
>>>>>> part of the description).
>>>>>>
>>>>>> Action (Subheading): Description of techniques for meeting the SC
>>>>>> (could be
>>>>>> techniques which are already in the techniques document or new
>>>>>> techniques
>>>>>> which are not in the document, but with which the SC can be met).
>>>>>> Here even
>>>>>> usability aspects can play a role, like: you can do a, b, c or d –
>>>>>> I/we
>>>>>> propose/recommend c.
>>>>>>
>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>
>>>>>> Result: pass/fail
>>>>>>
>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>> list of
>>>>>> pages where the problem exists
>>>>>>
>>>>>> Problem (Subheading): description/explanation of existing
>>>>>> problems and
>>>>>> barriers for users (here know how out of the Understanding Document
>>>>>> could
>>>>>> be
>>>>>> part of the description).
>>>>>>
>>>>>> Action (Subheading): Description of techniques for meeting the SC
>>>>>> (could be
>>>>>> techniques which are already in the techniques document or new
>>>>>> techniques
>>>>>> which are not in the document, but with which the SC can be met).
>>>>>> Here even
>>>>>> usability aspects can play a role, like: you can do a, b, c or d –
>>>>>> I/we
>>>>>> propose/recommend c.
>>>>>>
>>>>>> (...)
>>>>>>
>>>>>> ======
>>>>>>
>>>>>> Use cases:
>>>>>>
>>>>>> - test incl. consulting
>>>>>>
>>>>>> - for clients who are not very familiar with accessibility and WCAG2
>>>>>>
>>>>>> ============
>>>>>>
>>>>>> For a seal/badge or any formal confirmation Option 1 is the minimum.
>>>>>>
>>>>>> A report might also / should? also have intro parts like:
>>>>>>
>>>>>> - Short description of the Option 1, 2 or 3
>>>>>>
>>>>>> - Something like a disclaimer ("results might not be complete,
>>>>>> therefore it
>>>>>> is important to go through the page, view all similar elements and
>>>>>> solve
>>>>>> the
>>>>>> corresponding problems)
>>>>>>
>>>>>> - Glossary (for specific terms we used in our methodology -like
>>>>>> regional/global – if we decide to use them)
>>>>>>
>>>>>> - Documentation of the used OS, Browsers and Versions, probably used
>>>>>> assistive technologies incl. versions
>>>>>>
>>>>>> - Tested Conformance Level (A, AA, AA)
>>>>>>
>>>>>> - Results
>>>>>>
>>>>>> - Summary, probably written as an overall impression - we discussed
>>>>>> in this
>>>>>> list the 'motivation factor'. I think the aim of an evaluation is
>>>>>> not to
>>>>>> motivate. Nevertheless, writing a nice overall impression in a
>>>>>> report, may
>>>>>> have this function. Ok, except when there is nothing nice to say.
>>>>>>
>>>>>> This scheme could probably also be used for processes, pdf, flash
>>>>>> and
>>>>>> so on
>>>>>> and I think it would be flexible enough (time, costs, ...) and in
>>>>>> the
>>>>>> same
>>>>>> time valid against the Conformance Requirements, because the core
>>>>>> (evaluation itself) is the same in every option.
>>>>>>
>>>>>> Important, as I see it, is that the evaluator has the three
>>>>>> different
>>>>>> aspects in mind and in the report, which I believe shouldn't be
>>>>>> mixed:
>>>>>> evaluation (Core, testing SCs), explanation (description of the
>>>>>> problem/violation, understanding) and consulting (how to meet,
>>>>>> usability,..)
>>>>>>
>>>>>>
>>>>>> The evaluator could document the "progress toward meeting success
>>>>>> criteria
>>>>>> from all levels beyond the achieved level of conformance": If for
>>>>>> example
>>>>>> the evaluation is for Level A with Option 3 the SCs of AA could
>>>>>> also be
>>>>>> checked (pass/fail) without any further description or with further
>>>>>> description, depending on the contract.
>>>>>>
>>>>>> Advantage: every evaluator or testing organization uses the
>>>>>> methodology and
>>>>>> a standardized 'template' for the core and the evaluation itself.
>>>>>> The
>>>>>> descriptions of existing barriers (explanatory part/understanding in
>>>>>> Option
>>>>>> 2 and 3) and the consulting part (How to meet, in Option 3) would
>>>>>> be the
>>>>>> specific added value for the clients/the evaluator/the testing
>>>>>> organization.
>>>>>>
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Best
>>>>>>
>>>>>> --Kerstin
>>>>>>
>>>>>>
>>>>>> -------------------------------------
>>>>>> Kerstin Probiesch - Freie Beraterin
>>>>>> Barrierefreiheit, Social Media, Webkompetenz
>>>>>> Kantstraße 10/19 | 35039 Marburg
>>>>>> Tel.: 06421 167002
>>>>>> E-Mail: mail@barrierefreie-informationskultur.de
>>>>>> Web: http://www.barrierefreie-informationskultur.de
>>>>>>
>>>>>> XING: http://www.xing.com/profile/Kerstin_Probiesch
>>>>>> Twitter: http://twitter.com/kprobiesch
>>>>>> ------------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B

Received on Wednesday, 22 February 2012 11:33:51 UTC