Re: Step 1.e: Define the Techniques as used

Hi Richard, 

Reading 1e and re-reading it, in light of your comments… there do seem to be several nuances… 

You could say that the intension, in 1e in mentioning the W3C WCAG 2.0 Techniques, was for an evaluator to agree with the evaluation commissioner what Test/Procedure sections (relating to different Techniques) were to be followed in the evaluation - allowing the evaluation commissioner to come back and say - we actually followed these techniques for this SC, rather than the ones you have chosen to look at… - "to ensure consistent expectations".

If we extrapolate this to the nth degree you realise that the conversation between the evaluator and the evaluation commissioner might be a time-wasting exercise IF they insist you evaluate against the techniques they chose to use (the commissioner is after all paying you for a service). I admit, that you might need to evaluate additional things outside of their techniques selection - if for example you become aware that they have misunderstood things, or have looked too narrowly at SC...

But, on the other hand IF they say they have not followed any techniques (as Aurélien suggests) you would of course be free to evaluate whatever you see fit.  

Either way, I still think it is necessary to record what you actually assessed (and now I would advocate writing down why you chose to assess what you did e.g. the commissioner insisted due to x, no techniques were mentioned by the commissioner, etc...) - 100% for the sake of reproducibility.

Thanks to Richard for shedding additional light on 1.e.

Alistair

On 1 Jun 2012, at 16:52, RichardWarren wrote:

> Dear All,
> I think I may be partly to blame for some confusion here by discussing with Vivienne  SC2.4.1 (skip links) within this thread – sorry!
>  
> Regarding Section 3 step 1e  - I am of the opinion that requirement 1.e is a statement of intent.
>  
> Requirement 1.e: The techniques that will be used to evaluate the conformance to WCAG 2.0 should be specified.
>  
> In my opinion it refers to the techniques *used by the evaluator* to evaluate the website for conformance to WCAG 2.
> An example statement here might be :-
>  
> 1) An automated programme (Watchfire Bobby) will trawl the entire site to list missing required elements and required element attributes.
> 2) Experts will examine individual pages to check that suitable techniques have been used to ensure accessibility.
> 3) A panel of disabled users will attempt a series of suitable tasks (defined elsewhere)
> 4) Various browsers and Assistive technology tools will be used including – Internet Explorer (8), Firefox (11), Google Chrome (19), Safari (5), Jaws (11) and Zoomtext. 
>  
> As such I probably agree with Alistair that it is not optional. It is required if there is any query about the result of the evaluation because it allows all sides to be clear as to how the evaluation was conducted. This should also make the evaluation replicable.
>  
> Just to be clear – I do not think this step has anything to do with how the web-designer meets WCAG, so there is no need for a discussion on WCAG Success Criteria or WCAG techniques. It might be better if we reword the title of this requirement (3-1e) to better match the text introducing section 3.
>  
> In the introduction we use the words “process used by the evaluator”. Perhaps requirement 1e should read
>  
> Requirement 1.e: The processes that will be used to evaluate the conformance to WCAG 2.0 should be specified.
>  
> Kind regards
> Richard
>  
>  
> From: Alistair Garrison
> Sent: Friday, June 01, 2012 9:50 AM
> To: Detlev Fischer ; Eval TF
> Subject: Re: Step 1.e: Define the Techniques to be used
>  
> Hi Detlev, 
>  
> I accept your position with 1.e - and will remain firmly with mine (1.e. needed, and non-optional).  It would be interesting to hear what others think?
>  
> Also...
>  
> You write "You just check, in the set of selected UA, by tabbing, activating links, and so on if the page actually affords proper keyboard operability - not necessarily how, by which technique, this has been achieved."
>  
> I would just like to say that, to my mind, this again re-emphasises the need to test against web developer followed techniques - as otherwise you run the risk of missing things (technical accessibility evaluation should be supplemented by functional testing, not replaced).
>  
> You've picked an interesting example - which I think clearly highlights this very issue. If you take the following, an example only, but lets say its a wonderful function a keyboard user might really wish to use ;-)
>  
> function changeToBlue() {colorTarget.style.cssText = "colour:blue;";} (placed in script section)
>  
> <div id="colorTarget" onmouseover="changeToBlue()" onblur="changeToBlue()" style="color:red;">Test</div>
>  
> This element / function does not appear in the tab order - thinking about keyboard operability I believe it is a fail, but how would you know if you just check by tabbing?  This would, however, have been picked up if a web developer followed technique had been evaluated which said "all elements attached to JavaScript functionality appear in the tab order (or some such thing)" - it would also have been valuable for the web developer to know this.
>  
> Finally, I wish to clarify that once all the techniques followed by the developer have been evaluated I would whole-heartly suggest functional testing (more for usability for people with disabilities and realism) - but only afterwards, not instead.
>  
> All the best
>  
> Alistair
>  
> On 1 Jun 2012, at 09:49, Detlev Fischer wrote:
> 
>> Hi Alistair,
>> 
>> I'll leave it to others to reply as I have made my position clear (I hope). Just one clarification. You write:
>> 
>>> So, I'd really like to ask - "is 1.e being non-optional only objectionable to those who operate a third-party 'set in stone' evaluation methodology?" - as they could not possibly think of assessing anything other than they have already decided to assess…
>> 
>> 
>> I was trying to point out that the procedure must *not* be set in stone, i.e. check only for very specific techniques since it might miss other ways of meeting the Success Criteria. The assessment methods are often not tied to a particular technique anyway. If you check for keyboard operability, proper focus order, focus visibilty etc might have been created in a variety of ways. You just check, in the set of selected UA, by tabbing, activating links, and so on if the page actually affords proper keyboard operability - not necessarily how, by which technique, this has been achieved.
>> 
>> Regards,
>> Detlev
>> 
>> On 1 Jun 2012, at 08:42, Alistair Garrison wrote:
>> 
>>> Dear All,
>>> 
>>> The fact that evaluators each can have their own take even on WCAG 2.0 is proof enough that 1.e is needed - and should be non-optional.  If we fail to record the techniques selected and followed by the web developer, and ideally tested agains, what hope do we have of creating evaluations which are reproducible… or, indeed which provide useful feedback as to failings in implementation - and, importantly failings in technique selection.
>>> 
>>> To my mind, a goal of WCAG 2.0 is to let the web developer decide which techniques they believe satisfy the Success Criteria of WCAG 2.0 - and, the evaluators job to test what they have done.   That said, there will be a good deal of work ahead in educating people as to what satisfies each Success Criteria - and, 1.e being non-optional could play an important role.
>>> 
>>> On a different note - if we look at an evaluation carried out by the same company that creates the website (first party) it would be laughable if they tested against anything but the techniques their developers used to create the website - why would they…  For them, following 1.e would be a 'no-brainer'…
>>> 
>>> So, I'd really like to ask - "is 1.e being non-optional only objectionable to those who operate a third-party 'set in stone' evaluation methodology?" - as they could not possibly think of assessing anything other than they have already decided to assess…
>>> 
>>> Interested to hear any thoughts…
>>> 
>>> All the best
>>> 
>>> Alistair
>>> 
>>> On 1 Jun 2012, at 05:37, Vivienne CONWAY wrote:
>>> 
>>>> Hi Richard
>>>> I've actually been having a long debate (they get pretty excited) with the IG about this one.  According to WCAG 2, 2.4.1. is met if the ST of a correct heading structure is applied.  While I don't agree that having headings should be a sufficient technique on its own (due to the fact it doesn't help keyboard users), it appears to be set in WCAG 2 that way.  It fails 2.4.1 if there are no (working - implied I think) skip links and the heading structure is either non-existent or insufficient to bypass repeated navigation structures.
>>>> 
>>>> Your thoughts?
>>>> 
>>>> 
>>>> Regards
>>>> 
>>>> Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
>>>> PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
>>>> Director, Web Key IT Pty Ltd.
>>>> v.conway@ecu.edu.au
>>>> v.conway@webkeyit.com
>>>> Mob: 0415 383 673
>>>> 
>>>> This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
>>>> ________________________________________
>>>> From: RichardWarren [richard.warren@userite.com]
>>>> Sent: Friday, 1 June 2012 1:45 AM
>>>> To: detlev.fischer@testkreis.de; alistair.j.garrison@gmail.com; public-wai-evaltf@w3.org
>>>> Subject: Re: Step 1.e: Define the Techniques to be used
>>>> 
>>>> Hi Detlev,
>>>> 
>>>> As mentioned before, meeting one individual SC does not mean automatically
>>>> meeting the actual guideline subsection. In the case you mention - correct
>>>> semantics (headings) can provide a way for blind users to navigate more
>>>> easily (incl. skiping blocks). However a sighted keyboard user with a
>>>> standard browser does not usually have access to the semantic code in the
>>>> way that a screen reader does. So for these users we still need to provide a
>>>> "skip" link for long navigation lists at least.
>>>> 
>>>> So if "Commissioner says we have implemented skip links to meet 2.4.1 Bypass
>>>> Blocks"  then I say great, but you also need to have suitable heading codes
>>>> (and possibly something like "skip code samples" if the site is an on-line
>>>> course in HTML) so we will check that your site has mechanism/s for
>>>> bypassing repetitive blocks and non-informative blocks whilst we are at it
>>>> for compliance with guideline 2.4.1.
>>>> 
>>>> Richard
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: detlev.fischer@testkreis.de
>>>> Sent: Thursday, May 31, 2012 4:22 PM
>>>> To: alistair.j.garrison@gmail.com ; public-wai-evaltf@w3.org
>>>> Subject: Re: Fwd: Step 1.e: Define the Techniques to be used
>>>> 
>>>> Hi Alistair, hi all,
>>>> 
>>>> Don't know if it is a good idea to answer here since this now goes into the
>>>> "Disposition of Comments" but I'll have a go nevertheless.
>>>> 
>>>> As I understand it, we need to look for each SC if any of the Sufficient
>>>> Techniques (or a set of combined techniques as expressed in the options of
>>>> the "How to meet" document) has been suvessfully used. For that, it is not
>>>> sufficient to test techniques being put forward by the comissioner.
>>>> 
>>>> Example:
>>>> * Commissioner says "we have implemented skip links to meet 2.4.1 Bypass
>>>> Blocks"
>>>> * You evaluate and find that for some reason skip links aren't properly
>>>> implemented (fail of that technique)
>>>> * There is a proper headings structure that meets SC 4.2.1 (or ARIA
>>>> landmarks in a context where that is accessibility supported)
>>>> 
>>>> Now as long as you don't hit a failure, I guess you woud need to say pass to
>>>> the SC even though the technique submitted did not work.
>>>> (Having said that, the faulty skip links may fail other SC, but not SC
>>>> 2.4.1).
>>>> 
>>>> Any thoughts?
>>>> 
>>>> Regards,
>>>> Detlev
>>>> 
>>>> ----- Original Message -----
>>>> From: alistair.j.garrison@gmail.com
>>>> To: public-wai-evaltf@w3.org
>>>> Date: 31.05.2012 17:06:52
>>>> Subject: Fwd: Step 1.e: Define the Techniques to be used
>>>> 
>>>> 
>>>>> Dear All,
>>>>> 
>>>>> Would it be possible to add my comments about Step 1.e to the comments
>>>>> document - http://www.w3.org/WAI/ER/conformance/comments
>>>>> 
>>>>> Begin forwarded message:
>>>>> 
>>>>>> From: Alistair Garrison <alistair.j.garrison@gmail.com>
>>>>>> Subject: Step 1.e: Define the Techniques to be used
>>>>>> Date: 10 May 2012 10:48:41 CEST
>>>>>> To: Eval TF <public-wai-evaltf@w3.org>
>>>>>> 
>>>>>> Dear All,
>>>>>> 
>>>>>> "Step 1.e: Define the Techniques to be used" - could we consider making
>>>>>> this step non-optional?
>>>>>> 
>>>>>> The first reason being that we really need to check their implementation
>>>>>> of the techniques (W3C, their own code of best practice or whatever) they
>>>>>> say they use.
>>>>>> 
>>>>>> For example:
>>>>>> 
>>>>>> - Case 1) If they have done something by using technique A, and we
>>>>>> evaluate using technique B there could be an issue (they might fail B);
>>>>>> - Case 2) If they have done something by using technique A, and we
>>>>>> evaluate using technique A and B there still could be an issue (they
>>>>>> might fail B);
>>>>>> - Case 3) If they have done something by using technique A, and we
>>>>>> evaluate using technique A - it seems to work.
>>>>>> 
>>>>>> The second reason being that testing seems only to be really replicable
>>>>>> if we know what the techniques were they said they implemented -
>>>>>> otherwise, two different teams could easily get two different results
>>>>>> based on the cases above.
>>>>>> 
>>>>>> I would be interested to hear your thoughts.
>>>>>> 
>>>>>> Very best regards
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>> 
>>>> 
>>>> This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.
>>>> 
>>>> CRICOS IPC 00279B
>>>> 
>>>> 
>>> 
>> 
>> -- 
>> Detlev Fischer
>> testkreis - das Accessibility-Team von feld.wald.wiese
>> c/o feld.wald.wiese
>> Borselstraße 3-7 (im Hof)
>> 22765 Hamburg
>> 
>> Tel   +49 (0)40 439 10 68-3
>> Mobil +49 (0)1577 170 73 84
>> Fax   +49 (0)40 439 10 68-5
>> 
>> http://www.testkreis.de
>> Beratung, Tests und Schulungen für barrierefreie Websites
>> 
>> 
>> 
> 
>