Fwd: What happens when an evaluation statement already exists?

What timing… Seems they have similar issues with regard to monitoring tools ;-)

http://lists.w3.org/Archives/Public/public-auto-wcag/2014May/0000

Begin forwarded message:

> From: Alistair Garrison <alistair.j.garrison@gmail.com>
> Subject: Re: What happens when an evaluation statement already exists?
> Date: 19 May 2014 17:06:56 GMT+01:00
> To: Detlev Fischer <detlev.fischer@testkreis.de>, Eval TF <public-wai-evaltf@w3.org>
> 
> I develop web content / apps in addition to evaluating web content.  I therefore approach this work from a web developers point of view and an evaluators point of view, which may be the reason for our different perspectives. 
> 
> As such, I do not want to be shackled later in life by a document I helped to create - which is why I think this scenario is important.   Especially, in light of a coming EU directive which may well site this evaluation methodology as the cornerstone of on-going monitoring / conformance activities. 
> 
> If I say I have used this set of techniques, and I have undertaken all associated checks, I would not want to be told that I did not conform because I did not use another set of techniques - possibly bound together in some monitoring tool.  
> 
> Or, think about two EU states using slightly different monitoring tools - which conflict.   Having chaired the Evaluation Methods Task Force for the EuroAccessibility Consortium, and a similar European task force in the initial WABCluster I can say that such conflicts are the rule, rather than the exception. 
> 
> So, a concrete proposal for an addition to step 1.d would be:
> 
> "If documented evaluation procedures are provided along with an evaluation statement or WCAG 2.0 conformance claim, the documented evaluation procedures provided should take precedence over any other evaluation procedure when undertaking an evaluation."
> 
> Alistair
> 
> On 19 May 2014, at 16:16, Detlev Fischer wrote:
> 
>> 
>> On 19 May 2014, at 16:04, Alistair Garrison <alistair.j.garrison@gmail.com> wrote:
>> 
>>> Hi Detlev, 
>>> 
>>> How could you possibly maintain a procedure that checks for *all* a11y-supported ways - it would be ever expanding… it would also mean that you changed procedure often which would mean retesting often.  I don't think this is workable.
>> 
>> Well, for many aspects (like data tables) there is a finite number of ways to mark them up properly. It is true that a relevant procedure will at times be expanding in some direction (but also contracting by dropping obsolete techniques). In my view, it is the combination of techniques in a step-by-step procedure which brings value to testers - the BITV-Test has worked quite successfully along those lines for more than a decade.
>>> 
>>> I don't see why it is not surely easier to take on board the hard work of the website owners who have basically told you what they have checked… as I suggested, taking this onboard might only need a little phrase to be added to step 1.d.
>> 
>> I have no problem with the idea of picking up documented website owner techniques and using them to verify a claim. My point is merely that this scenario is not the usual one (looking at the usual testing requests we deal with). I think you have been the only guy in EVAL TF who has repeatedly brought up this scenario (which of course is not meant to suggest that it is an invalid one). For all I know, the situation in the UK may be different from what is our common experience. If an extra phrase covers this use case, fine.
>> 
>>> It is also true to say that the WCAG 2.0 techniques contain the test procedures which would need to be followed, so I don't understand your last point about "WCAG-oriented checkpoints or checking procedures" - in fact, what are "WCAG-oriented checkpoints or checking procedures"? 
>> 
>> My point was that in all cases where the documented technique falls short somehow, it alone would not be sufficient to verify WCAG conformance or state that the SC has failed. You would in those cases look at other (known, feasible) ways to conform to the SC. Say, site owner tells you "we have ascertained that all text has a contrast ratio above 4,5:1" Now you found text below that threshold, so the site owner has failed his documented technique. But then you might discover a style switcher to load a CSS with better contrast - SC met, even if the style switcher was not used to claim conformance. The upshot is: The tester has to be open-minded and not just reproduce the techniques documented by the site owner. For that open-mindedness, a procedure aggregating applicable commonly found techniques to meet a SC in some logic has a value as a heuristic - especially for all people that are not already experts, but learn the ropes.
>> 
>> Hope that clarifies my response!
>> 
>> Best, 
>> Detlev
>>> 
>>> All the best 
>>> 
>>> Alistair
>>> 
>>> On 19 May 2014, at 14:44, Detlev Fischer wrote:
>>> 
>>>> Hi Alistair,
>>>> 
>>>> not following a particular technique where alternatives sufficient techniques exist would never be a reason to fail sites in a WCAG-Test. If a monitoring project said that according to their evaluation your website fails, it may not strictly follow the WCAG approach.  It may fail your site according to their particular test procedure, not according to WCAG. (In this way, we at BITV-Test still require text-only magnification and good graphics contrast, knowing that regarding these aspects we currently go beyond the minimum requirenments of WCAG).
>>>> 
>>>> So in my view, a WCAG checkpoint for correctly marked-up data tables (to take your example) would need a procedure that checks for *all* a11y-supported ways of marking up the table properly, and pass the content if any sufficient technique (or custom technique that has been demonstrated to meet the SC) has been used. I guess even ARIA role="rowheader" and role="columnheader" might qualify (at least in constrained environments when it can be shown that the selected set of UA and AT supports them).
>>>> 
>>>> That's why WCAG-oriented checkpoints or checking procedures will always be different from particular techniques (WCAG or other) - they focus at a certain SC (or a particular aspect of an umbrella SC like 1.3.1, lke table mark-up) not a particular technique. And they will need maintenance over time to include coverage of new approaches / techniques to meet the SC.
>>>> 
>>>> Best,
>>>> Detlev
>>>> 
>>>> --
>>>> Detlev Fischer
>>>> testkreis c/o feld.wald.wiese
>>>> Thedestr. 2, 22767 Hamburg
>>>> 
>>>> Mobil +49 (0)1577 170 73 84
>>>> Tel +49 (0)40 439 10 68-3
>>>> Fax +49 (0)40 439 10 68-5
>>>> 
>>>> http://www.testkreis.de
>>>> Beratung, Tests und Schulungen für barrierefreie Websites
>>>> 
>>>> Alistair Garrison schrieb am 19.05.2014 12:14:
>>>> 
>>>>> Dear All, 
>>>>> 
>>>>> Let's say I'm a website owner and I follow specific techniques in order to
>>>>> create a website which, as far as I know, is compliant.  I make a claim to this
>>>>> effect, after evaluating the website and creating an evaluation statement
>>>>> (which details which checks I have done, from the techniques I have followed). 
>>>>> 
>>>>> 
>>>>> Now let's say a national monitoring project checks my website, but their
>>>>> conformance model is based on a different set of techniques, and of course
>>>>> checks.  
>>>>> 
>>>>> After their checks the monitoring project say that according to their
>>>>> evaluation my website fails - maybe one or two things.  Really only due to the
>>>>> mismatch between techniques e.g. I used the headers attribute technique to make
>>>>> tables compliant, whilst they only checked for scope, etc…
>>>>> 
>>>>> Would I have to change my website? This seems a little silly… 
>>>>> 
>>>>> My question is then - what happens when a proper evaluation statement already
>>>>> exists?
>>>>> 
>>>>> I think we have spoken around this subject a number of times, but there is no
>>>>> clear advice that I can find in the methodology.
>>>>> 
>>>>> My hope would be that if the evaluation statement has been made properly and is
>>>>> up-to-date, any further evaluations undertaken (with or without the knowledge
>>>>> of the website owner) should in some way respect the techniques followed and
>>>>> the way in which the page has been evaluated - especially if the further
>>>>> evaluations are to be used for monitoring, etc…
>>>>> 
>>>>> Otherwise, I worry that the techniques and their checks selected in outside
>>>>> evaluations, especially monitoring evaluations, will start to constrain the
>>>>> techniques which developers can follow - which is at odds with WCAG 2.0. 
>>>>> 
>>>>> I would suggest that the most likely area for discussion on this subject is
>>>>> step 1.d.
>>>>> 
>>>>> Anyway, interested to hear your thoughts and comments.
>>>>> 
>>>>> All the best 
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> -- 
>> Detlev Fischer
>> testkreis - das Accessibility-Team von feld.wald.wiese
>> c/o feld.wald.wiese
>> Thedestraße 2
>> 22767 Hamburg
>> 
>> Tel   +49 (0)40 439 10 68-3
>> Mobil +49 (0)1577 170 73 84
>> Fax   +49 (0)40 439 10 68-5
>> 
>> http://www.testkreis.de
>> Beratung, Tests und Schulungen für barrierefreie Websites
>> 
>> 
>> 
>> 
>