AW: AW: Additional Point/Question: problem centered / page centered evaluation

Hi Detlev, TF,

first I want to say sorry: I tried to be brief, but...
 
> I would have thought that
> the existence of WCAG success criteria means to check pages or
> processes *across the board* to to spot and record *any* problems that
> may exist, also less obvious ones. Problems become apparent only
> during those checks (if they are not glaringly obvious  - but even
> then, we must look out also for less obvious ones).

In my understanding of WCAG 2.0 the existence of the Conformance Requirements, especially Conformance Level

"1. Conformance Level: One of the following levels of conformance is met in full.
 * Level A: For Level A conformance (the minimum level of conformance), the Web page satisfies all the Level A Success Criteria, or a conforming alternate version is provided.
 * Level AA: For Level AA conformance, the Web page satisfies all the Level A and Level AA Success Criteria, or a Level AA conforming alternate version is provided.
 * Level AAA: For Level AAA conformance, the Web page satisfies all the Level A, Level AA and Level AAA Success Criteria, or a Level AAA conforming alternate version is provided."

means that when testing a single page (that one the mentioned web developer has added) or a process a tester must check all SCs on the page resp. on every page of a process. The question for is: Is the Conformance Level mandatory for the evaluation methodology of a website itself or just for the result? Checking all pages of a given website is no problem in the case of small websites. It is also no problem when testing a huge website: with a lot of time, patience, a huge team and of course a lot of money (depending on the number of pages).

In case of websites I'm considering about something like this - in analogy to the Conformance Level:

"1. Conformance Level: One of the following levels of conformance is met in full.
 * Level A: For Level A conformance (the minimum level of conformance), all Web pages satisfy all the Level A Success Criteria, or for those Web pages which don't satisfy a conforming alternate version is provided.
 * Level AA: For Level AA conformance, all Web pages satisfy all the Level A and Level AA Success Criteria, or a Level AA conforming alternate version is provided for those Web pages which don't satisfy all the Criteria.
(...)?

Cause the website owner wants to know or have to know (e.g. because of legal justifications) if the website (not only one or some pages) satisfies all the Level A Success Criteria and probably also Level AA. The question is: how do we come to an result which reflects the "reality" of the given website.

> Maybe I misunderstood. Can please someone step forward and explain
> what a "problem-centred" approach is supposed to mean and how it can
> ensure that important barriers are not overlooked?

I try to go more in detail and of course according my understanding of the two methodologies: 

- Page centered - just as I understand it:
Following this it is necessary to define not only the amount of tested pages but also the type (typical, common...). According to this evaluation methodology the tester has to find those pages which should be tested and which are the typical or common (whatever common means) pages. After he/she will test every SC on every page of the 5, 8, 9, 10 or more? pages. The more pages a tester checks the more the final result(s) will be reliable and reflects the "reality". But even then: he/she will never know - as long as he/she tested every page. This might not be tricky for some barriers but it is very tricky for others, especially the already mentioned SCs 1.3.1, 1.3.3 and others.

With page centered the question is: are there violations on the X checked pages? This might be sufficient for some SCs. The question is: is it sufficient for all? 

- problem centered
Barriers can be on every single page, on some pages and just one page (probably this page is not a "common" one). There might be violations of SCs on every single page, some pages and just one. Some examples and considerations about this point to explain better what problem centered _could_ mean. Guided by the SCs he/she will check if there are videos (mostly they are easy to find with the website search), he/will check if there are forms and so on and if there are barriers according to the SCs. The check of some SCs could start parallel. With activated HeadingsMap for example a tester sees the heading structure of every visited single page during the whole testing procedure in a parallel window. Another example: If 3.1.1 is violated it will be violated in most cases on every single page. If 1.3.3 is violated the barrier might be on some pages or just one page. Guided by a problem centered evaluation methodology the tester can check typical phrases ("click on the right button",...,...). In a lot of cases he/she can use the website site search or google ("click on the right button" site:www.website.com). Every finding could be a violation of the SC (the tester has to check if so) and could mean that Level A is not met for one page or some or many. What we have to describe and also find out - if we as TF will recommend a problem centered methodology (I think this is also just one example for problem-centered) - are intelligent methods to check every SC on as many of pages as "possible" in an "acceptable" time. Page-centered would mean, that the tester will check 1.3.3 just on X pages. He/she will never know if there is a violation against 1.3.3.

I'm not uncritical at this point. The evaluator needs a lot of experience and he has to document all the findings in a good way. And of course an evaluation methodology like this is more strict than testing X pages.
 
> So I think if we want to distinguish between the common page or
> page-sample pased approach and something else, this something else
> would be *processes*: a particular sequence of pages, page states and
> user interactions that lead to a definable result. One can focus on
> those, but we should keep in mind that the accessibility of individual
> pages must not be ignored over this. So a problem for any calculation
> of score is how the assessment of processes can be integrated with
> results from page sample checks which in my view won't simply be
> unnecessary - unless you define the use of *any* page alone as a
> process...

"Calculation of score" or binary (pass/fail) is one of the open questions. I would prefer to discuss this separately. Thoughts?

Thinking about different situations:

- website test
- comparative studies
- short reviews
- single pages
- processes

probably we as TF should/could? recommend both evaluation methodologies depending on "what we want to know": A comparative study might be well done with page-centered evaluation methodologies. For the case of the website owner who wants to know if his website reaches Conformance Level A, AA or AAA I'm in doubt if checking X pages will be sufficient - especially in the light of Conformance Level: _met in full_ and _satisfies all the Level A, AA, AAA Criteria_.

> how it (problem-centered) can
> ensure that important barriers are not overlooked?

I think this a general question with a lot of variables: discipline, experience, knowledge about tools. When tester checks the SCs for videos he has to look if there are videos. He/she must have all the relevant SCs in mind (incl. possible keyboard problems), the same with forms. Probably he/she will not start with SC 1.1.1 but with 1.3.1 or first the videos. And for sure there will be SC-sequences which will be more helpful in finding barriers than other sequences. Problem-centered may look in the beginning as if there are just mosaic pieces or as unsystematic ad-hoc testing but after a check the tester has a very good overview. Unless he/she is documenting the findings well the result will be an evaluation report with descriptions of the founded barriers incl. a documentation of the checked pages (incl. the URI) and a statement (Conformance Level).

I hope this is/was helpful for the discussion.

Regards

Kerstin


> -----Ursprüngliche Nachricht-----
> Von: public-wai-evaltf-request@w3.org [mailto:public-wai-evaltf-
> request@w3.org] Im Auftrag von fischer@dias.de
> Gesendet: Montag, 5. September 2011 09:41
> An: public-wai-evaltf@w3.org
> Betreff: Re: AW: Additional Point/Question: problem centered / page
> centered evaluation
> 
> Hi EVAL TF,
> 
> To me, it is not at all clear what a "problem-centred" approach to web
> site accessibility evaluation actually is. I would have thought that
> the existence of WCAG success criteria means to check pages or
> processes *across the board* to to spot and record *any* problems that
> may exist, also less obvious ones. Problems become apparent only
> during those checks (if they are not glaringly obvious  - but even
> then, we must look out also for less obvious ones).
> 
> 
> Maybe I misunderstood. Can please someone step forward and explain
> what a "problem-centred" approach is supposed to mean and how it can
> ensure that important barriers are not overlooked?
> 
> Regards,
> Detlev
> 
> Quoting kvotis@iti.gr:
> 
> > Hi to all
> >
> > i aggree with Richard and others regarding the problem-centred
> approach.
> > We have followed the same approach in the FP7 ACCESSIBLE project
> > assessment tools where the evaluation could be also performed through
> the
> > usage of different categories of personas, functional limitations,
> etc.
> >
> > regards
> >
> > kostas
> >
> >
> > -------------------
> > Dr. Konstantinos Votis
> > Computer Engineer & Informatics,PhD, Msc, MBA
> > Research Associate
> > Informatics and Telematics Institute
> > Centre for Research and Technology Hellas
> > 6th Klm. Charilaou - Thermi Road
> > P.O. BOX 60361 GR - 570 01
> > Thessaloniki – Greece
> > Tel.: +30-2311-257722
> > Fax : +30-2310-474128
> > E-mail : kvotis@iti.gr
> >
> >
> >
> >> Hi,
> >> I agree with Vivienne and Kerstin that a problem-centred approach is
> best.
> >> Using problems means that you look at the whole website (if
> appropriate)
> >> and
> >> are able to identify particular groups that are disadvantaged by the
> >> issues
> >> found. This is much more useful if you want to compare sites.
> Explaining
> >> that 20% of pages are not accessible has less impact than saying
> that a
> >> particular group (keyboard users,  blind users etc.) cannot use the
> site.
> >>
> >> You can still apply the problem-centred approach to single pages,
> >> templates
> >> or components.
> >>
> >> Richard
> >>
> >> -----Original Message-----
> >> From: Kerstin Probiesch
> >> Sent: Saturday, September 03, 2011 12:08 PM
> >> To: 'Eval TF'
> >> Cc: 'Vivienne CONWAY'
> >> Subject: AW: Additional Point/Question: problem centered / page
> centered
> >> evaluation
> >>
> >> Hi Vivienne, hi TF,
> >>
> >>> The question of problem-centered or page-centered approach is going
> to
> >>> be tricky.
> >>
> >> Yes. It is.
> >>
> >>> Personally I favour the problem-centered approach as I am
> >>> seldom assessing a single page.  Violations in my opinion, usually
> are
> >>> duplicated across pages.  For example if there is a lack of
> keyboard
> >>> access on a single page, that is likely to occur on most if not all
> >>> pages.  Some websites are so huge, that a page-centered approach is
> >>> simply not feasible.
> >>
> >> Same opinion. When following a page-centered approach you have to
> document
> >> the same problem on every page again and again. Keyboard access is a
> good
> >> example for that, another one is SC 3.1.1. The more single pages you
> have,
> >> the more the page-centered approach might be imo insufficient. Not
> only
> >> for
> >> the accessibility statement/report/result but also for the steps a
> website
> >> owner has to do or order to do. An example: you have a huge website
> with
> >> hundreds or more single pages and - let me say - 10 or more
> categories. In
> >> this case an evaluation guided by a problem-centered approach will
> find
> >> more
> >> violations and according to my experiences you can e.g. find out if
> >> all/not
> >> all/just some/or one online editor/s know how to organize the
> content (Hx,
> >> lists,...) or how to deal with SC 1.1.1. In addition you can give
> specific
> >> instructions.
> >>
> >> Another problem of the page-centered approach is imo the following:
> You
> >> _have_ to define a set of "typical" pages to test on every website
> as part
> >> of the methodology. A website owner would complain if you test
> different
> >> types. What happens in following situation: on one homepage (main
> page)
> >> you
> >> have a video which violates the SC, on another website you also have
> a
> >> video
> >> which violates the SC but the video is not on the main page and
> probably
> >> also not on one of the defined set of pages according to the
> followed
> >> page-centered approach? One can argue: in this case we are testing
> also
> >> the
> >> video. But, what about 1.3.3, 1.4.1 and 1.3.1 and how to deal with
> >> processes? Important questions I think are
> >>
> >> - are accessible statements which are reliable and valid based upon
> X
> >> (3,4,5,10?) pages possible and how many pages one have to test? This
> >> corresponds with "R04: Replicability: different Web accessibility
> >> evaluators
> >> who perform the same tests on the same site should get the same
> results
> >> within a given tolerance."
> >> - how avoid human errors during the process of page selection before
> >> starting a particular test and during the test itself?
> >>
> >> Nevertheless and even if I prefer a problem-centered approach, a
> >> page-centered one is probably the only acceptible and pragmatical
> method
> >> when you have to test a great amount of websites in a comparative
> study -
> >> but here again I think we will have limitations with consequences
> for the
> >> accessibility statement(s).
> >>
> >> I suggest that we also discuss, if we as TF should give
> redommendations/
> >> suggestions for different scenarios:
> >>
> >> - testing one single page (test should include every SC on this
> single
> >> page)?!
> >> - testing a website (walkthrough for every single SC _and_ every
> single
> >> step
> >> of a process, if existing)?!
> >> - comparative studies (testing every SC on a defined set of
> "typical"
> >> pages)
> >> ?!
> >> - (...)?!
> >>
> >> And we also have to consider the conformity levels.
> >>
> >> I'm thinking about if it might be helpful to collect possible
> scenarios to
> >> see which approach might be the best for a given scenario resp.
> discuss if
> >> we recommend one approach for all scenarios?
> >>
> >> Last not least: Also the problem-centered approach might lead to
> problems.
> >> In the page-centered approach a tester has finished the test when
> checked
> >> all SCs for the collected single pages. In a problem-centered
> approach one
> >> have to find other criterias otherwise the tester is "lost in
> violations"
> >> depending on the quality of the website.
> >>
> >>
> >>> On the other hand, a developer who is adding a single page and
> wanting
> >>> to know if it meets accessibility criteria would be testing only
> that
> >>> page.  Thoughts?
> >>
> >> This is an interesting question. It can be that the added single
> page
> >> belongs to an already tested website or not. If the added single
> belongs
> >> to
> >> an already tested site, maybe the test is long ago. I think in case
> of an
> >> added single page a test should include all SCs which indicates the
> >> problem-centered approach.
> >>
> >> I hope my points are clear. Sadly, I'm not very familiar in writing
> >> English
> >> :-( practice will do, hopefully.
> >>
> >> Regards
> >>
> >> Kerstin
> >>
> >>
> >>>
> >>>
> >>> Regards
> >>>
> >>> Vivienne L. Conway
> >>> ________________________________________
> >>> From: public-wai-evaltf-request@w3.org [public-wai-evaltf-
> >>> request@w3.org] On Behalf Of Shadi Abou-Zahra [shadi@w3.org]
> >>> Sent: Tuesday, 30 August 2011 5:05 PM
> >>> To: Eval TF
> >>> Subject: Fwd: Additional Point/Question: problem centered / page
> >>> centered  evaluation
> >>>
> >>> -------- Original Message --------
> >>> Subject: Additional Point/Question: problem centered / page
> centered
> >>> evaluation
> >>> Date: Tue, 30 Aug 2011 08:21:34 +0000
> >>> From: Kerstin Probiesch <mail@barrierefreie-informationskultur.de>
> >>> To: <public-wai-evaltf@w3.org>
> >>>
> >>> Dear Eval TF,
> >>>
> >>> I want to add the following question: should we recommend a problem
> >>> centered
> >>> or a page centered approach? In my opinion this question is not
> only
> >>> essential for further discussions about single questions and topics
> >>> (true/false, rankings, involve people with disabilities) but
> essential
> >>> for
> >>> the whole methodology.
> >>>
> >>> Regs
> >>>
> >>> Kerstin
> >>>
> >>>
> >>> -------------------------------------
> >>> Kerstin Probiesch - Freie Beraterin
> >>> Barrierefreiheit, Social Media, Webkompetenz
> >>> Kantstra?e 10/19 | 35039 Marburg
> >>> Tel.: 06421 167002
> >>> E-Mail: mail@barrierefreie-informationskultur.de
> >>> Web: http://www.barrierefreie-informationskultur.de
> >>>
> >>> XING: http://www.xing.com/profile/Kerstin_Probiesch
> >>> Twitter: http://twitter.com/kprobiesch
> >>> ------------------------------------
> >>>
> >>> This e-mail is confidential. If you are not the intended recipient
> you
> >>> must not disclose or use the information contained within. If you
> have
> >>> received it in error please return it to the sender via reply e-
> mail
> >>> and delete any record of it from your system. The information
> contained
> >>> within is not the opinion of Edith Cowan University in general and
> the
> >>> University accepts no liability for the accuracy of the information
> >>> provided.
> >>>
> >>> CRICOS IPC 00279B
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
>

Received on Thursday, 8 September 2011 08:05:38 UTC