RE: Thoughts about the differentiation of reporting types (Basic | Detailed | In-depth)

Hi Detlev,

Good input! We added this section because of the need defined by EvalTF for different levels of reporting and the possible impact on the evaluation (time) effort. I applaud your proposal to rethink this section. But I would like to start that after we publish the current version as a Public Working Draft. Hope that is ok? 

Would you then be willing to propose an alternative setup for these goals that we can discuss on the list and Telco (but after publication of the current version)? We will then have time to discuss that while the Public Review period is on.

If EvalTF is ok and you are ok with it, I can log it as task in the tracker.
Kindest regards,

Eric

________________________________________
Van: detlev.fischer@testkreis.de [detlev.fischer@testkreis.de]
Verzonden: woensdag 5 september 2012 14:47
Aan: public-wai-evaltf@w3.org
Onderwerp: Thoughts about the differentiation of reporting types (Basic | Detailed |  In-depth)

Hi all,

Going through the new draft of WCAG-EM, I began to wonder whether we should differentiate Basic Report and Detailed Report a bit more, leading to thoughts whether we really need all three types, and in what sense they differ.

Here is the text quoted from 3.5.1 Step 5.a: Provide Documentation for Each Step:

* Basic Report
    "Only captures the successes and failures in meeting WCAG 2.0 conformance requirements globally for the entire website . For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target , the report identifies if it is met or not met in the selected sample of web pages . Where failures in meeting WCAG 2.0 Success Criteria are identified, each web page in which such a failure has been identified must be indicated in the report."

* Detailed Report
    "Captures the successes and failures in meeting WCAG 2.0 conformance requirements for each web page . For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target , the report identifies if it is met or not met in each web page in the selected sample of web pages . Where failures in meeting WCAG 2.0 Success Criteria on a web page are identified, each identified occurrence of such a failure must be indicated in the report."

It seems that while Basic Report lists SC and for each of these, failed pages, the Detailed Report lists sampled pages and for each of these, failed SC. The real difference is the requirement in the Detailed Report to list *each instance*, which could easily get very tedious and counter-productive (just think of a longish text using <br> instead of <p>). Would you really want a list of all these violations of SC 1.3.1 in a Detailed Report? Maybe we should change to "each identified occurrence of such a failure, or each type of failure where occuring instances are repetitive, must be indicated in the report."

The wording in  3.5.1 Step 5.a: to describe the types of report may be changed to acknowledge that the two ways of reporting are basically homologous:
* listing all pages (with failures) in the sample, then listing all failed SC per page
* listing all WCAG SC on the chosen level of conformance, then listing all pages that failed

I think we should not mandate either way of sorting results in Basic Report and Detailed Report; both can be mapped on the respective other or changed through some DB sort command in an evaluation tool. The way it reads now, it appears as if providing SC first, then failed pages (Basic Report) is something quite different from providing pages first, then SCs that failed (Detailed Report).

The *real* difference between the two types of report is the requirement to enlist *all* the failure instances in the Detailed Report. If this is not done in a mechanical way (e.g., by providing line numbers, which BTW may still not be sufficiently accurate), it requires a comment identifying where and how a SC failed - which is getting close to the In-Depth report.

The only added value I can currently see in the Detailed report is getting some rough quantitative measure of the number of instances where a failure occured (which can be quite a misleading indicator without referencing criticality of the failure).

Regards,
Detlev
--
Detlev Fischer
testkreis c/o feld.wald.wiese
Borselstraße 3-7 (im Hof), 22765 Hamburg

Mobil +49 (0)1577 170 73 84
Tel +49 (0)40 439 10 68-3
Fax +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites

Received on Wednesday, 5 September 2012 13:54:24 UTC