Re: Not Applicable (was Re: Evaluation scheme with three options - proposal)

Hi Vivienne and all.

When I use the "pass" term for things that do not exist I get queries from 
my clients who think that I have not done the job properly.

To save us all the hassle of explaining the complexity of W3C logic,  I 
therefore use N/A for SCs and Guideline 1.2.  on my score sheet.  At the top 
of my score sheet I explain the meanings of Pass, Fail and N/A including 
that we give a pass score value to N/A  because the designer has avoided 
using the relevant technologies that can cause particular problems. This 
means the in the executive summary I can count N/As as Pass and avoid the 
hassle. (eg. "The site passes all 12 guidelines")

Richard

-----Original Message----- 
From: Vivienne CONWAY
Sent: Thursday, February 23, 2012 9:09 AM
To: Shadi Abou-Zahra
Cc: Eval TF
Subject: RE: Not Applicable (was Re: Evaluation scheme with three options - 
proposal)

HI Shadi & all

That's true, however where it comes unstuck is in the reporting.  CEO's like 
to see lovely graphs showing how many of the criteria have been met.  If the 
data is recorded as 'passed' even though it doesn't exist (the audio file), 
then the graph and any stats look better than they are really are.  If it is 
n/a or something similar, they can't confuse this with something that 
actually meets a requirement.


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual 
or entity named above. If you are not the intended recipient, you are 
notified that any dissemination, distribution or copying of this email is 
strictly prohibited. If you have received this email in error, please notify 
me immediately by return email or telephone and destroy the original 
message.
________________________________________
From: Shadi Abou-Zahra [shadi@w3.org]
Sent: Thursday, 23 February 2012 6:02 PM
To: Vivienne CONWAY
Cc: Eval TF
Subject: Re: Not Applicable (was Re: Evaluation scheme with three options - 
proposal)

Hi Vivienne,

Isn't it the same if you called it "passed" or "not applicable" then the
content is added? In both cases your report is already out of date and
the content needs to be reassessed.

Note that even in this most basic report it is pretty clear when the
content passed because there was no corresponding content:
  - <http://www.w3.org/WAI/demos/bad/before/reports/home.html>

Best,
   Shadi


On 23.2.2012 09:47, Vivienne CONWAY wrote:
> HI Alistair
> I agree with you on this one for sure.
>
> If I can't find an issue (say an audio file) that does not necessarily 
> mean that the website should pass on that criteria. It may be added later, 
> or I may just not have located it when looking for suitable pages to 
> assess.  I'm more comfortable with 'not applicable' or 'not tested' or 
> 'not located on reviewed pages' or something similar.
>
>
> Regards
>
> Vivienne L. Conway, B.IT(Hons), MACS CT
> PhD Candidate&  Sessional Lecturer, Edith Cowan University, Perth, W.A.
> Director, Web Key IT Pty Ltd.
> v.conway@ecu.edu.au<mailto:v.conway@ecu.edu.au>
> v.conway@webkeyit.com<mailto:v.conway@webkeyit.com>
> Mob: 0415 383 673
>
> This email is confidential and intended only for the use of the individual 
> or entity named above. If you are not the intended recipient, you are 
> notified that any dissemination, distribution or copying of this email is 
> strictly prohibited. If you have received this email in error, please 
> notify me immediately by return email or telephone and destroy the 
> original message.
>
> ________________________________
> From: Alistair Garrison [alistair.j.garrison@gmail.com]
> Sent: Thursday, 23 February 2012 5:43 PM
> To: Eval TF
> Subject: Re: Not Applicable (was Re: Evaluation scheme with three 
> options - proposal)
>
> Hi All,
>
> I would feel a little uncomfortable declaring something to be passed - 
> simply because I could not find any applicable content.
>
> I know when showing conformity with ISO documentation "reference to the 
> appropriate Standard (title, number, and year of issue); when the 
> certification applies to only a Portion of a standard, the applicable 
> portion(s) should be clearly identified;" (ISO Guide 23).
>
> In our situation, this would mean that we simply list all things (most 
> probably Success Criteria) we have found to be applicable in the 
> Conformance Claim.  Then go on to state which of these things has been 
> passed or failed in the report.
>
> Hope this helps.
>
> All the best
>
> Alistair
>
> On 22 Feb 2012, at 12:37, Emmanuelle Gutiérrez y Restrepo wrote:
>
> My +1 too :-)
>
> I think that this is very important.
>
> Regards,
> Emmanuelle
>
> 2012/2/22 Velleman, 
> Eric<evelleman@bartimeus.nl<mailto:evelleman@bartimeus.nl>>
> Hi Vivienne,
>
> Think I put it on the agenda. So lets talk about it.
> Kindest regards,
>
> Eric
>
>
>
> ________________________________________
> Van: Vivienne CONWAY [v.conway@ecu.edu.au<mailto:v.conway@ecu.edu.au>]
> Verzonden: woensdag 22 februari 2012 3:13
> Aan: Michael S Elledge; Shadi Abou-Zahra
> CC: public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>
> Onderwerp: RE: Not Applicable (was Re: Evaluation scheme with three 
> options  -  proposal)
>
> Hi all
>
> I just ran across this discussion which is something that I think we 
> should put in the EVTF methodology.  I know that I've been using n/a when 
> it seemed the item was not present in the website e.g. videos.  However if 
> this is the W3C consensus, I'll need to change my reporting.  Can we talk 
> about this in our teleconference this week?
>
>
> Regards
>
> Vivienne L. Conway, B.IT<http://B.IT/>(Hons), MACS CT
> PhD Candidate&  Sessional Lecturer, Edith Cowan University, Perth, W.A.
> Director, Web Key IT Pty Ltd.
> v.conway@ecu.edu.au<mailto:v.conway@ecu.edu.au>
> v.conway@webkeyit.com<mailto:v.conway@webkeyit.com>
> Mob: 0415 383 673
>
> This email is confidential and intended only for the use of the individual 
> or entity named above. If you are not the intended recipient, you are 
> notified that any dissemination, distribution or copying of this email is 
> strictly prohibited. If you have received this email in error, please 
> notify me immediately by return email or telephone and destroy the 
> original message.
> ________________________________________
> From: Michael S Elledge [elledge@msu.edu<mailto:elledge@msu.edu>]
> Sent: Wednesday, 22 February 2012 2:29 AM
> To: Shadi Abou-Zahra
> Cc: public-wai-evaltf@w3.org<mailto:public-wai-evaltf@w3.org>
> Subject: Re: Not Applicable (was Re: Evaluation scheme with three 
> ptions  - proposal)
>
> Thanks for the explanation, Shadi. I imagine it took some discussion to
> reach that consensus!  :^)
>
> Mike
>
> On 2/20/2012 2:30 PM, Shadi Abou-Zahra wrote:
>> Hi Mike,
>>
>> Good question. We had a long discussion about that and also asked the
>> WCAG Working Group on their position on this.
>>
>> According to WCAG WG, the term "Not Applicable" is not defined and is
>> ambiguous. Accessibility requirements are deemed met when the content
>> does not require specific accessibility features. For example, the
>> requirement for captioning is deemed met if there is no video content.
>>
>> I will try dig out the corresponding pointers but I recall that this
>> was something that was less clearly documented in the WCAG documents.
>> We will probably need to clarify this point somewhere in section 5 of
>> the Methodology, and possibly as WCAG WG to also clarify some of their
>> materials (so that we can refer to it from our explanation).
>>
>> Best,
>>    Shadi
>>
>>
>> On 20.2.2012 19:26, Michael S Elledge wrote:
>>> Hi Shadi--
>>>
>>> I noticed in the BAD example that success criteria for which there was
>>> no related web content received a "Pass." I'm curious why that approach
>>> was chosen rather than to identify such instances as "Not Applicable" or
>>> "NA." Wouldn't using the term "NA" be both more informative and
>>> accurate?
>>>
>>> Mike
>>>
>>> On 2/20/2012 10:43 AM, Shadi Abou-Zahra wrote:
>>>> Small addition:
>>>>
>>>> On 20.2.2012 16:28, Shadi Abou-Zahra wrote:
>>>>> Hi Kerstin, All,
>>>>>
>>>>> I'm not too sure what the difference between options #1 and #2
>>>>> would be
>>>>> in practice, as I hope that evaluators will simply link to Techniques
>>>>> rather than to attempt to explain the issues themselves.
>>>>>
>>>>>
>>>>> Here is an example of what a report of option #1 could look like:
>>>>> -<http://www.w3.org/WAI/demos/bad/before/reports/home.html>
>>>>
>>>> Here is a positive example too: ;)
>>>> -<http://www.w3.org/WAI/demos/bad/after/reports/home.html>
>>>>
>>>>
>>>> Regards,
>>>> Shadi
>>>>
>>>>
>>>>> Note: this is a report for a single page but it could still be a basis
>>>>> for reports of option #1 for entire websites; it just has a pass/fail
>>>>> for each Success Criterion and some Techniques to justify these
>>>>> claims.
>>>>>
>>>>>
>>>>> For option #2 we could introduce a scoring function in addition to the
>>>>> pass/fail result. This would require the evaluators to fully evaluate
>>>>> every page in the selected sample and count the frequencies of
>>>>> errors to
>>>>> calculate a score. It could help compare websites and motivate the
>>>>> developers (at least those who are close to full compliance).
>>>>>
>>>>>
>>>>> Finally, option #3 would be more in-depth reports with examples of the
>>>>> errors and explanations of ways to repair the errors. These are, as
>>>>> Kerstin says, developed by consultants (as opposed to pure evaluators)
>>>>> for developers who are new to accessibility.
>>>>>
>>>>> We attempted to provide such an example report in the initial
>>>>> version of
>>>>> the Before and After Demo (BAD) but it is really lots of work:
>>>>> -<http://www.w3.org/WAI/EO/2005/Demo/report/>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Shadi
>>>>>
>>>>>
>>>>> On 19.2.2012 20:36, Elle wrote:
>>>>>> Kerstin:
>>>>>>
>>>>>> I like these three options. I am interested, however, in how many
>>>>>> clients
>>>>>> that typically ask for something as abbreviated as Option 1. For
>>>>>> those in
>>>>>> this group, do you experience situations with a lot of clients who
>>>>>> don't
>>>>>> want more than the pass/fail report?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Elle
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 4:36 AM, Kerstin Probiesch<
>>>>>> k.probiesch@googlemail.com<mailto:k.probiesch@googlemail.com>> 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> in our last teleconference we discussed a evaluation scheme with
>>>>>>> three
>>>>>>> options based upon 100% Conformance. I appreciate these proposals
>>>>>>> and
>>>>>>> see
>>>>>>> them as chance to integrate or point to the three documents of
>>>>>>> WCAG2:
>>>>>>> Guidelines and SCs, Understanding and How to meet.
>>>>>>>
>>>>>>> One proposal for handling the documents in an evaluation scheme,
>>>>>>> based upon
>>>>>>> the normative guidelines and SCs as core:
>>>>>>>
>>>>>>> =====
>>>>>>> Option 1: WCAG 2.0 – Core Test ("light version" or whatever the
>>>>>>> wording
>>>>>>> later will be)
>>>>>>>
>>>>>>> # Guideline X (Heading)
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) - – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) - – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> (...)
>>>>>>>
>>>>>>> =====
>>>>>>>
>>>>>>> Use cases for Option1:
>>>>>>>
>>>>>>> - experienced developers and clients who know WCAG2 and need just
>>>>>>> the
>>>>>>> results,
>>>>>>> - comparative evaluations (20 hotel websites, city websites…)
>>>>>>> - or for example just with the SCs of level a and a smaller scope as
>>>>>>> pre-test to decide together with the client what the best next steps
>>>>>>> might
>>>>>>> be (evaluation, consulting, probably workshops for editors)
>>>>>>>
>>>>>>> =====
>>>>>>>
>>>>>>> Option 2: WCAG 2.0 – Core incl. understanding (name?)
>>>>>>>
>>>>>>> # Guideline X (Heading)
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> Problem (Subheading): Description of existing problems and barriers
>>>>>>> for
>>>>>>> users (here know how out of the understanding document could be part
>>>>>>> of the
>>>>>>> description).
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> Problem (Subheading): Description of existing problems and barriers
>>>>>>> for
>>>>>>> users (here know how out of the understanding document could be part
>>>>>>> of the
>>>>>>> description).
>>>>>>>
>>>>>>> (...)
>>>>>>>
>>>>>>> ======
>>>>>>>
>>>>>>> Use cases:
>>>>>>>
>>>>>>> - comparative evaluations (depending on the specific time and costs)
>>>>>>>
>>>>>>> - if a client just want descriptions
>>>>>>>
>>>>>>> - regular tests like "evaluation of the week"
>>>>>>>
>>>>>>> =====
>>>>>>>
>>>>>>> Option 3: WCAG 2.0 – Core, understanding, how to meet (name?)
>>>>>>>
>>>>>>> # Guideline X (Heading)
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> Problem (Subheading): description/explanation of existing
>>>>>>> problems and
>>>>>>> barriers for users (here know how out of the Understanding Document
>>>>>>> could
>>>>>>> be
>>>>>>> part of the description).
>>>>>>>
>>>>>>> Action (Subheading): Description of techniques for meeting the SC
>>>>>>> (could be
>>>>>>> techniques which are already in the techniques document or new
>>>>>>> techniques
>>>>>>> which are not in the document, but with which the SC can be met).
>>>>>>> Here even
>>>>>>> usability aspects can play a role, like: you can do a, b, c or d –
>>>>>>> I/we
>>>>>>> propose/recommend c.
>>>>>>>
>>>>>>> ## Checkpoint: SC XX (Subheading)
>>>>>>>
>>>>>>> Result: pass/fail
>>>>>>>
>>>>>>> Character: global/regional (or another wording) – if regional: a
>>>>>>> list of
>>>>>>> pages where the problem exists
>>>>>>>
>>>>>>> Problem (Subheading): description/explanation of existing
>>>>>>> problems and
>>>>>>> barriers for users (here know how out of the Understanding Document
>>>>>>> could
>>>>>>> be
>>>>>>> part of the description).
>>>>>>>
>>>>>>> Action (Subheading): Description of techniques for meeting the SC
>>>>>>> (could be
>>>>>>> techniques which are already in the techniques document or new
>>>>>>> techniques
>>>>>>> which are not in the document, but with which the SC can be met).
>>>>>>> Here even
>>>>>>> usability aspects can play a role, like: you can do a, b, c or d –
>>>>>>> I/we
>>>>>>> propose/recommend c.
>>>>>>>
>>>>>>> (...)
>>>>>>>
>>>>>>> ======
>>>>>>>
>>>>>>> Use cases:
>>>>>>>
>>>>>>> - test incl. consulting
>>>>>>>
>>>>>>> - for clients who are not very familiar with accessibility and WCAG2
>>>>>>>
>>>>>>> ============
>>>>>>>
>>>>>>> For a seal/badge or any formal confirmation Option 1 is the minimum.
>>>>>>>
>>>>>>> A report might also / should? also have intro parts like:
>>>>>>>
>>>>>>> - Short description of the Option 1, 2 or 3
>>>>>>>
>>>>>>> - Something like a disclaimer ("results might not be complete,
>>>>>>> therefore it
>>>>>>> is important to go through the page, view all similar elements and
>>>>>>> solve
>>>>>>> the
>>>>>>> corresponding problems)
>>>>>>>
>>>>>>> - Glossary (for specific terms we used in our methodology -like
>>>>>>> regional/global – if we decide to use them)
>>>>>>>
>>>>>>> - Documentation of the used OS, Browsers and Versions, probably used
>>>>>>> assistive technologies incl. versions
>>>>>>>
>>>>>>> - Tested Conformance Level (A, AA, AA)
>>>>>>>
>>>>>>> - Results
>>>>>>>
>>>>>>> - Summary, probably written as an overall impression - we discussed
>>>>>>> in this
>>>>>>> list the 'motivation factor'. I think the aim of an evaluation is
>>>>>>> not to
>>>>>>> motivate. Nevertheless, writing a nice overall impression in a
>>>>>>> report, may
>>>>>>> have this function. Ok, except when there is nothing nice to say.
>>>>>>>
>>>>>>> This scheme could probably also be used for processes, pdf, flash
>>>>>>> and
>>>>>>> so on
>>>>>>> and I think it would be flexible enough (time, costs, ...) and in
>>>>>>> the
>>>>>>> same
>>>>>>> time valid against the Conformance Requirements, because the core
>>>>>>> (evaluation itself) is the same in every option.
>>>>>>>
>>>>>>> Important, as I see it, is that the evaluator has the three
>>>>>>> different
>>>>>>> aspects in mind and in the report, which I believe shouldn't be
>>>>>>> mixed:
>>>>>>> evaluation (Core, testing SCs), explanation (description of the
>>>>>>> problem/violation, understanding) and consulting (how to meet,
>>>>>>> usability,..)
>>>>>>>
>>>>>>>
>>>>>>> The evaluator could document the "progress toward meeting success
>>>>>>> criteria
>>>>>>> from all levels beyond the achieved level of conformance": If for
>>>>>>> example
>>>>>>> the evaluation is for Level A with Option 3 the SCs of AA could
>>>>>>> also be
>>>>>>> checked (pass/fail) without any further description or with further
>>>>>>> description, depending on the contract.
>>>>>>>
>>>>>>> Advantage: every evaluator or testing organization uses the
>>>>>>> methodology and
>>>>>>> a standardized 'template' for the core and the evaluation itself.
>>>>>>> The
>>>>>>> descriptions of existing barriers (explanatory part/understanding in
>>>>>>> Option
>>>>>>> 2 and 3) and the consulting part (How to meet, in Option 3) would
>>>>>>> be the
>>>>>>> specific added value for the clients/the evaluator/the testing
>>>>>>> organization.
>>>>>>>
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> --Kerstin
>>>>>>>
>>>>>>>
>>>>>>> -------------------------------------
>>>>>>> Kerstin Probiesch - Freie Beraterin
>>>>>>> Barrierefreiheit, Social Media, Webkompetenz
>>>>>>> Kantstraße 10/19 | 35039 Marburg
>>>>>>> Tel.: 06421 167002
>>>>>>> E-Mail: 
>>>>>>> mail@barrierefreie-informationskultur.de<mailto:mail@barrierefreie-informationskultur.de>
>>>>>>> Web: 
>>>>>>> http://www.barrierefreie-informationskultur.de<http://www.barrierefreie-informationskultur.de/>
>>>>>>>
>>>>>>> XING: http://www.xing.com/profile/Kerstin_Probiesch
>>>>>>> Twitter: http://twitter.com/kprobiesch
>>>>>>> ------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> This e-mail is confidential. If you are not the intended recipient you 
> must not disclose or use the information contained within. If you have 
> received it in error please return it to the sender via reply e-mail and 
> delete any record of it from your system. The information contained within 
> is not the opinion of Edith Cowan University in general and the University 
> accepts no liability for the accuracy of the information provided.
>
> CRICOS IPC 00279B
>
>
>
>
>
>
>
> --
> Emmanuelle Gutiérrez y Restrepo
> Fundación y Seminario SIDAR
> URL: www.sidar.org<http://www.sidar.org/>
> email: emmanuelle@sidar.org<mailto:emmanuelle@sidar.org>
>
>
> ________________________________
> This e-mail is confidential. If you are not the intended recipient you 
> must not disclose or use the information contained within. If you have 
> received it in error please return it to the sender via reply e-mail and 
> delete any record of it from your system. The information contained within 
> is not the opinion of Edith Cowan University in general and the University 
> accepts no liability for the accuracy of the information provided.
>
> CRICOS IPC 00279B
>
>
>

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

This e-mail is confidential. If you are not the intended recipient you must 
not disclose or use the information contained within. If you have received 
it in error please return it to the sender via reply e-mail and delete any 
record of it from your system. The information contained within is not the 
opinion of Edith Cowan University in general and the University accepts no 
liability for the accuracy of the information provided.

CRICOS IPC 00279B

Received on Thursday, 23 February 2012 10:53:05 UTC