Re: Alternative concise requirements

Dear Shadi,

Thank you for your comments. Two heads are definitely better than one and I 
fully agree with all your suggestions.  I am particularly glad to get rid of 
the word "Unambiguous" <G>.

Kind regards
Richard

-----Original Message----- 
From: Shadi Abou-Zahra
Sent: Tuesday, October 04, 2011 6:05 AM
To: Richard Warren
Cc: Eval TF
Subject: Re: Alternative concise requirements

Hi Richard,

Thank you for your contribution, I like the approach a lot. I agree that
several requirements seem related and sometimes even overlap. I have not
checked back to see if we dropped anything but have some comments on
your suggestion:


On 4.10.2011 02:13, RichardWarren wrote:
> Hi,
> I am a little concerned that we are making life needlessly difficult by 
> having produced eighteen requirements for a methodology to evaluate 
> websites against twelve guidelines!
>
> Having read through the proposed requirements carefully I notice that some 
> are very similar (such as supporting verification and supporting validity, 
> or using unambiguous language and making it translatable).  Others are 
> confusing, such as saying it will not stretch into techniques and tests 
> (R01) and yet use existing testing techniques (R06).
>
> So I sat down and had a go at rationalising what we had produced and came 
> up with ten requirements that cover (I hope) all the points raised in our 
> various discussions. I am sure that it is not a perfect set of 
> requirements, but I would like to share it with you just in case you think 
> it might be the basis for a more concise list of requirements.
>
> --------------------------------------------------------------
>
> RQ 01 : Define methods for evaluating WCAG 2.0 conformance
> The Methodology provides methods to measure conformance with WCAG 2.0. 
> that can be used by the target audience (see section 2 above) for 
> evaluating small or large websites, sections of websites or web-based 
> applications.

Minor: "for evaluating small or large websites, sections of websites and
web-based applications" (changed "or" to "and").


> RQ 02 – Unambiguous Interpretation
> The methodology is written in clear language, understandable to the target 
> audience and capable of translation to other languages.

I think the title "Unambiguous Interpretation" does not match the
description. Maybe something like "Clear, understandable, and
translatable language" instead?


> RQ 03 – Reliable
> Different Web accessibility evaluators using the same methods on the same 
> website(s) should get the same results. Evaluation process and results are 
> documented to support independent verification.

Maybe "equivalent results" rather than "*same* results"?


> RQ 04 - Tool and browser independent
> The use and application of the Methodology is vendor-neutral and 
> platform-independent. It is not restricted to solely manual or automated 
> testing but allows for either or a combination of approaches.

I think we need to clarify "vendor-neutral" and "platform-independent".
I also think that the Methodology as a whole will have to rely on a
combined manual and automated approach. My suggestion is:

[[
The use and application of the Methodology is independent of any
particular evaluation tools, browsers, and assistive technology. It
requires combined use of manual and automated testing approaches to
carry out a full evaluation according to the Methodology.
]]


> RQ 05 -  QA framework specification guidelines
> The Methodology will conform to the Quality Assurance framework 
> specification guidelines as set in: http://www.w3.org/TR/qaframe-spec/.
>
> RQ 06 - Machine-readable reporting
> The Methodology includes recommendations for harmonized (machine-readable) 
> reporting. It provides a format for delivering machine-readable reports 
> using Evaluation and Report Language (EARL) in addition to using the 
> standard template as at http://www.w3.org/WAI/eval/template.html

I think that the focus on human-readable reporting is more important
than on machine-readable ones. Here is my suggestion:

[[
RQ 06 - Reporting
The Methodology includes recommendations for reporting evaluation
findings. It will be based on the
[href=http://www.w3.org/WAI/eval/template.html standard template] and
supplemented with machine-readable
[href=http://www.w3.org/WAI/intro/earl reports using Evaluation and
Report Language (EARL)].
]]


> RQ 07 -  Use of existing WCAG 2.0 techniques
> Wherever possible the Methodology will employ existing testing procedures 
> in the WCAG 2.0 Techniques documents rather than replicate them.
>
> RQ 08 -  Recommendations for scope and sampling
> It includes recommendations for methods of sampling web pages in large 
> websites and how to ensure that complete processes (such as for a shopping 
> site where all the pages that are part of the steps in an ordering 
> process) are included.  Such selections would be reflected in any 
> conformance claim.

Minor: I stumbled over "large" -- is a website with say 50 or 100 pages
considered large? It would still need sampling to evaluate...


> RQ 09 -  Includes tolerance metrics
> It includes calculation methods for determining nearness of conformance. 
> Depending on the amount of tolerance, a failure could fall within a 
> certain tolerance level meaning that the page or website might be 
> considered conformant even though there is a failure. Such tolerances 
> would be reflected in any conformance claim.
>
> RQ 10 - Support documentation
> The document will give a short description of the knowledge necessary for 
> using the Methodology for evaluations.

Thanks,
   Shadi


> Regards
> Richard

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Tuesday, 4 October 2011 09:39:06 UTC