Role of Sufficient Techniques in the methodology (was: Re: proposed definition for "website")

hi Kerstin, hi List,

I think it is important to clarify the role of the Sufficient Techniques 
in the Evaluation Methodology. Kerstin, you have pointed out a statement 
in the techniques document that says:

   "Test procedures are provided in techniques to help verify that
   the technique has been properly implemented."
   (...)
   "In particular, test procedures for individual techniques should
   not be taken as test procedures for the WCAG 2.0 success criteria
   overall."

 From this, you conclude that the Methodology should not refer to 
techniques at all. I think you may be misinterpreting the quoted 
statement. It does not warn against the use of techiques; it merely 
emphasises that any "fail" result when testing a particular technique 
does not show that the corresponding Success Criterion has not been met, 
because another Technique might have done the job.

The whole point of the "Sufficient Techniques" in WCAG ist to collect 
well-known ways of meeting Success Criteria and also offering tests that 
prove that the technique has been used correctly. The Quickref shows 
(sets of) Sufficient Techniques that can be used to meet SC. If the test 
of a Sufficient Technique (or all tests within the set of related 
techniques that together constitute one option for meeting the SC) 
returns a "pass", then the Success Criterion has been met.

Therefore, the Sufficient Techniques are a useful component for 
conducting the Evaluation, and this should be stated in Part 5.3 of the 
Methodology. While we will not deal with every SC on a detail level, I 
think it it is fair to state on a general level that the test procedure 
should check whether the "Sufficient Techniques"-options laid out in the 
Quickref ("How to meet WCAG 2.0") have been used to meet the SC under test.

Having said that, Section 5.3 should also state that if none of the 
documented Sufficient Techniques has been used, the SC might still have 
been met by an as yet undocumented technique. If they *are* successfully 
used, however, the SC is met.

What I am yet not sure about is whether section 5.3 should also include 
the application of the documented Failures related to a SC, or whether 
these are always the inverse of documented Sufficient Techniques. While 
many will be redundant, some may not be.

Regards,
Detlev





Am 08.02.2012 22:27, schrieb Michael S Elledge:
> Hi everyone--
>
> I don't think this is a problem so long as conformance is defined as
> meeting the full set of success factors, excepting those that are not
> applicable to the page (such as blinking content).
>
> It will always be up to the evaluator to decide which techniques to use
> to evaluate compliance with success factors, won't it?
>
> Mike
>
> On 2/8/2012 3:07 PM, Kerstin Probiesch wrote:
>> Hi all,
>>
>> this would mean: testing techniques and not SCs.
>>
>> Another problem is suitable" especially when - like proposed here -
>> the testing organization defines the techniques (subset). This will
>> lead to different subsets (techniques) and to different results
>> depending on what different testing organizations or testers define as
>> "suitable".
>>
>> -- Kerstin
>>
>>
>>
>> 2012/2/8 Detlev Fischer<fischer@dias.de>:
>>> I think any full evaluation would have to include the full set of SC
>>> for a
>>> conformance claim, but it is still useful to be able to deploy a
>>> suitable
>>> subset, for example, to evaluate the output of a particular portlet. The
>>> definition of the subset would be down to the organization doing the
>>> testing. Such a portlet test would be no more than a building block
>>> towards
>>> a final full evaluation (or may serve its purpose outside the context
>>> of a
>>> full WCAG conformance claim). At least that's the way I see it...
>>>
>>> Regards,
>>> Detlev
>>>
>>>
>>> to Quoting Elle<nethermind@gmail.com>:
>>>
>>>> Detlev:
>>>>
>>>> Does this mean we would theoretically have a full set and a sub set for
>>>> the
>>>> methodology?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Elle
>>>>
>>>>
>>>>
>>>> On Wed, Feb 8, 2012 at 12:15 PM, Detlev Fischer<fischer@dias.de> wrote:
>>>>
>>>>> Hi Don, hi list,
>>>>>
>>>>> I agree that the evaluation of portlets is an important use case. It
>>>>> raises the issue to what extent the methodology would support an
>>>>> evaluation
>>>>> based on a subset of WCAG success criteria.
>>>>>
>>>>> Just to be clear: I think a proper conformance statement can only
>>>>> be made
>>>>> on the level of the full web page. The testing of a number of success
>>>>> criteria depends not only on the assessment of the full page (as
>>>>> for most
>>>>> SC within principle 2: Operable), but beyond that, on the page
>>>>> *context*,
>>>>> as in 2.4.5 Multiple Ways, 3.2.3 Consistent Navigation, and 3.2.4
>>>>> Consistent Identification. And some success criteria (or subsets
>>>>> thereof,
>>>>> such as 1.3.1: headings) may or may not apply on the level of
>>>>> portlet-generated code fragment, depending on the approach. For
>>>>> example,
>>>>> in
>>>>> an HTML 5 context, the headings hierarchy may be generated from the
>>>>> nesting
>>>>> of fragments so having all portals use h1 might be fine.
>>>>>
>>>>> Having said that, it is clearly useful to be able to evaluate
>>>>> portlets or
>>>>> other code framents (whatever way they are generated) with an
>>>>> applicable
>>>>> subset of WCAG Success Criteria. This would give organizations a
>>>>> means to
>>>>> measure the degree of conformance on a unit level long before a final
>>>>> conformance claim or test. The full conformance statement would
>>>>> then only
>>>>> be made on the level of the portal page.
>>>>>
>>>>> How much of the methodology is actually useful for a selective
>>>>> evaluation
>>>>> based on a subset of WCAG success criteria needs to be carefully
>>>>> looked
>>>>> at.
>>>>>
>>>>> Regards,
>>>>> Detlev
>>>>>
>>>>>
>>>>> e.g. Quoting Shadi Abou-Zahra<shadi@w3.org>:
>>>>>
>>>>>
>>>>> Hi Don,
>>>>>>
>>>>>> I would say that this depends on what the scope of the evaluation is,
>>>>>> which needs to be clearly stated in the conformance statement/report.
>>>>>>
>>>>>> It could make sense to have each portlet evaluated separately, though
>>>>>> this would not include the main portal entry page and similarly pages
>>>>>> that
>>>>>> belong to the portal but are not specific to individual portals.
>>>>>>
>>>>>> Actually, I think this is an important use-case for us to consider.
>>>>>>
>>>>>> Best,
>>>>>> Shadi
>>>>>>
>>>>>>
>>>>>> On 3.2.2012 17:40, Don Raikes wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Maybe I have missed a discussion of this topic since I just
>>>>>>> joined the
>>>>>>> group, but what about a portal-style site?
>>>>>>>
>>>>>>> What if one or more portlets on the site are accessible, but the
>>>>>>> container portal page has some accessibility issues how do we handle
>>>>>>> this?
>>>>>>> Also what about the fact that some portlets are accessible and
>>>>>>> others
>>>>>>> are
>>>>>>> not? Do we consider each portlet a website since it is a
>>>>>>> self-contained
>>>>>>> web
>>>>>>> application?
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Shadi Abou-Zahra [mailto:shadi@w3.org]
>>>>>>> Sent: Friday, February 03, 2012 4:38 AM
>>>>>>> To: Eval TF
>>>>>>> Subject: proposed definition for "website"
>>>>>>>
>>>>>>> Dear Eval TF,
>>>>>>>
>>>>>>> Yesterday I took an action to propose a new definition for "website"
>>>>>>> *in
>>>>>>> the context of this document*. Currently the document says:
>>>>>>>
>>>>>>> [[
>>>>>>> http://www.w3.org/WAI/ER/**methodology/#website<http://www.w3.org/WAI/ER/methodology/#website>
>>>>>>>
>>>>>>>
>>>>>>> Website - A coherent collection of one or more related web pages
>>>>>>> that
>>>>>>> together provide common use or functionality. It includes static web
>>>>>>> pages,
>>>>>>> dynamically generated web pages, and web applications.
>>>>>>> ]]
>>>>>>>
>>>>>>>
>>>>>>> I propose we add the following text to this definition:
>>>>>>>
>>>>>>> [[
>>>>>>> Websites are generally self-enclosed entities with key resources
>>>>>>> such
>>>>>>> as
>>>>>>> a homepage, login page, or other entry pages; help pages;
>>>>>>> sitemap; and
>>>>>>> contact information. Web pages within a website typically have a
>>>>>>> common
>>>>>>> design ("look and feel") and navigational structures. Examples of
>>>>>>> websites
>>>>>>> in the context of this document include:
>>>>>>> - Self-enclosed web appearances owned or provided by organizations;
>>>>>>> - Self-enclosed sections of websites (sometimes referred to as
>>>>>>> "sub-sites"), such as these of departments within organizations;
>>>>>>> - Self-enclosed web applications and other web-based products.
>>>>>>>
>>>>>>> Arbitrary selections of individual web pages, especially those
>>>>>>> that do
>>>>>>> not include complete processes, are not regarded as websites.
>>>>>>>
>>>>>>> Note: Selections of individual web pages that are not regarded as
>>>>>>> websites may claim conformance to WCAG, but the evaluation of such
>>>>>>> collections is outside the scope of this methodology.
>>>>>>> ]]
>>>>>>>
>>>>>>>
>>>>>>> Looking forward to your reactions!
>>>>>>>
>>>>>>> Best,
>>>>>>> Shadi
>>>>>>>
>>>>>>> --
>>>>>>> Shadi Abou-Zahra -
>>>>>>> http://www.w3.org/People/**shadi/<http://www.w3.org/People/shadi/>Activity
>>>>>>>
>>>>>>> Lead, W3C/WAI International Program Office Evaluation and Repair
>>>>>>>
>>>>>>> Tools Working Group (ERT WG) Research and Development Working Group
>>>>>>> (RDWG)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Shadi Abou-Zahra -
>>>>>> http://www.w3.org/People/**shadi/<http://www.w3.org/People/shadi/>
>>>>>>
>>>>>> Activity Lead, W3C/WAI International Program Office
>>>>>> Evaluation and Repair Tools Working Group (ERT WG)
>>>>>> Research and Development Working Group (RDWG)
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ------------------------------**------------------------------**---
>>>>>
>>>>> Detlev Fischer PhD
>>>>> DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
>>>>> Geschäftsführung: Thomas Lilienthal, Michael Zapp
>>>>>
>>>>> Telefon: +49-40-43 18 75-25
>>>>> Mobile: +49-157 7-170 73 84
>>>>> Fax: +49-40-43 18 75-19
>>>>> E-Mail: fischer@dias.de
>>>>>
>>>>> Anschrift: Schulterblatt 36, D-20357 Hamburg
>>>>> Amtsgericht Hamburg HRB 58 167
>>>>> Geschäftsführer: Thomas Lilienthal, Michael Zapp
>>>>> ------------------------------**------------------------------**---
>>>>>
>>>>>
>>>>
>>>> --
>>>> If you want to build a ship, don't drum up the people to gather wood,
>>>> divide the work, and give orders. Instead, teach them to yearn for the
>>>> vast
>>>> and endless sea.
>>>> - Antoine De Saint-Exupéry, The Little Prince
>>>>
>>>
>>>
>>> --
>>> ---------------------------------------------------------------
>>> Detlev Fischer PhD
>>> DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
>>> Geschäftsführung: Thomas Lilienthal, Michael Zapp
>>>
>>> Telefon: +49-40-43 18 75-25
>>> Mobile: +49-157 7-170 73 84
>>> Fax: +49-40-43 18 75-19
>>> E-Mail: fischer@dias.de
>>>
>>> Anschrift: Schulterblatt 36, D-20357 Hamburg
>>> Amtsgericht Hamburg HRB 58 167
>>> Geschäftsführer: Thomas Lilienthal, Michael Zapp
>>> ---------------------------------------------------------------
>>>
>>
>>
>

Received on Thursday, 9 February 2012 10:32:02 UTC