W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > June 2012

RE: AW: Aspects of an evaluator

From: Vivienne CONWAY <v.conway@ecu.edu.au>
Date: Mon, 25 Jun 2012 08:36:28 +0800
To: RichardWarren <richard.warren@userite.com>, Kerstin Probiesch <k.probiesch@googlemail.com>, 'Alistair Garrison' <alistair.j.garrison@gmail.com>, 'Eval TF' <public-wai-evaltf@w3.org>
Message-ID: <8AFA77741B11DB47B24131F1E38227A9C29DAD5E8E@XCHG-MS1.ads.ecu.edu.au>
Hi all
Following this discussion, I agree that we have set the stage for the Methodology to include a variety of reasons for testing and persons who would be interested in both the testing and results.

My opinion, is that we do not include 'independent' in this section, and that we address the issue of the compliance claim in that section.  After all, anyone can decide to evaluate a website and we can't stipulate how independent or not they must be.  We can though in the compliance statement spell out that in order for a compliance statement to have worth, it must clearly state who has done the evaluation (whether person or company), and whether they are part of the commissioning organisation.

I often think of evaluations done by a company on their own website rather like a student composing an exam, taking the exam, and then marking it himself- reliability of his mark is rather 'a little' suspect.  However there is scope for this type of testing as well - for instance just as with a company preparing for a financial audit - they conduct an in-house audit first so that they can clear up what they can and make the job of the external auditor quicker and less costly.  In this case, the methodology can be used (and to be best prepared for the external audit, it should be), and then the same methodology used by the external auditor.  This would confirm that they have fixed the issues they found in their initial audit and give even more reliability to the final result.


Regards

Vivienne L. Conway, B.IT(Hons), MACS CT, AALIA(cs)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify me immediately by return email or telephone and destroy the original message.
________________________________________
From: RichardWarren [richard.warren@userite.com]
Sent: Monday, 25 June 2012 2:03 AM
To: Kerstin Probiesch; 'Alistair Garrison'; 'Eval TF'
Subject: Re: AW: Aspects of an evaluator

Dear Kerstin,
I earn my living as an independent evaluator. But that does not mean that I
have to follow a different methodology to an in-house evaluator. Hopefully
we will all use the same (this) methodology. The only time my independence
counts is at the point of issuing a compliance statement. When we get to
that part of the methodology we might consider how to phrase the statement
( i.e "Our in-house evaluators have issued a..." or "Independent evaluators
have issued a ..")

Until we get to the compliance statement part we should not add restrictions
to the position of evaluator as described in our terms of reference and
target audience.

As mentioned by Shadi - "trust and responsibility is more related to
certification schemes rather than to the evaluation procedure."

Section 3.1 is therefore not the place to add the adjective "independent",
or otherwise clarify the role of the evaluator.

Richard


-----Original Message-----
From: Kerstin Probiesch
Sent: Sunday, June 24, 2012 4:34 PM
To: 'Alistair Garrison' ; 'RichardWarren' ; 'Eval TF'
Subject: AW: Aspects of an evaluator

Hi all,

I suggest we collect those cases where a concept of an independent evaluator
will word and require it for those cases. I think an independent evaluator
is especially for third party evaluations important. We don't know all
testing organizations and it might be that an evaluator is also web
developer / agency. In this case the developer or agency is in reality a
first party but may act as third party.

And then we had also some discussion of a "second independent tester". If
the tester is from the same testing organization he might test a website
independent from the first tester, but this doesn't make him an independent
tester.

Cheers

Kerstin

Von: Alistair Garrison [mailto:alistair.j.garrison@gmail.com]
Gesendet: Freitag, 22. Juni 2012 11:22
An: RichardWarren; Eval TF
Betreff: Re: Aspects of an evaluator

Hi Richard,

We all know what we are doing is for 1st and 3rd Parties - and of course
they each have an equal weight.

The debate stems from Eric's questionnaire number 2 - question 6.

DoC_ID_28: Independent evaluator

I would say that I'm hearing most people are now against the proposed action
of stating that an evaluator must be independent (me too after Peter's
email) - as it would not work in all cases.

All the best

Alistair

On 22 Jun 2012, at 11:04, RichardWarren wrote:


If I might summarise our understanding of the role etc. of the evaluator

1) Peter has reminded us of our terms of reference and that it includes
reference to "different contexts such as self-assessment or third party
evaluation"

2) Our target audience (Section 1.2) includes -
Website developers, suppliers, procurers, and owners.
Web accessibility consultants and evaluation service providers.
Web accessibility monitoring and benchmarking activities.
Web accessibility researchers and disability advocates.

Having established the breadth of our target audience at the beginning we
cannot, half-way through our method, suddenly introduce the concept of only
allowing "independent"  or "third-party" evaluators.

As I understand it our task is to deliver a methodology that helps more
people to check if websites comply with WCAG so that the web as a whole
becomes more accessible.

The issue of the validity of conformance claims (transparency etc.) should
be a separate discussion

Regards
Richard



-----Original Message----- From: Alistair Garrison
Sent: Friday, June 22, 2012 8:28 AM
To: kvotis@iti.gr ; Peter Korn ; Eval TF
Subject: Re: Aspects of an evaluator

Dear Eval,

So, if we don't want do set requirements of independence on the evaluator,
and don't want to publish the procedures followed (or any parts of an
evaluation report) - how do we ensure public trust in the conformance
statement being made?

All the best

Alistair

On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote:


Dear Alistair,

regarding your following comment:
3) Insist that some parts of every report are made public - not
urls, just procedures (as it should be the case that any similar
samples of web pages taken from the defined site will lead to pretty
much the same result)


i am not sure about the described procedure and actually regarding  the
selection of appropriate parts of Web sites..How someone who is also
developer will select which of the parts should be selected?Moreover i
agree with Peter regarding the publicity of all evaluation reports. From
my point of view this is a difficult procedure while it requires heavy
storage and processing mechanisms

regards

Kostas



-------------------
Dr. Konstantinos Votis
Computer Engineer & Informatics,PhD, Msc, MBA
Research Associate
Informatics and Telematics Institute
Centre for Research and Technology Hellas
6th Klm. Charilaou - Thermi Road
P.O. BOX 60361 GR - 570 01
Thessaloniki &#8211; Greece
Tel.: +30-2311-257722
Fax : +30-2310-474128
E-mail : kvotis@iti.gr




Alistair,

I don't see how we can insist that all evaluation reports be public, or
that certain parts of an evaluation report be public.  We can say that
an evaluation report isn't complete unless it has all of the mandatory
parts.  But we can't stop someone from producing a summary of the
report, or excerpting parts of a report, or producing a description of
the report, or...

Not unless we require copyright on all reports generated by anyone else
and license it and...  (and even then "fair use" doctrines in many
countries would likely still allow publishing excerpts).


Peter


On 6/21/2012 1:31 PM, Alistair Garrison wrote:
Hi Peter,

Trust issues aside? I'd be really interested to hear your thoughts on
the 3rd option I presented?

All the best

Alistair


On 21 Jun 2012, at 22:26, Peter Korn wrote:

Alistair,

I think the question of trust is, frankly, outside of the scope of
our work as well.  Nothing prevents someone from lying - whether or
not they are independent.  They may make honest mistakes because they
lack the technical expertise needed to do a good job (again whether
independent or not). They may claim their sample is representative
but it isn't. They may claim something failed when it passed (or
vice-versa).

Being independent doesn't prevent any of that.


Regards,

Peter

On 6/21/2012 12:35 PM, Alistair Garrison wrote:
Hi Peter,

Your point about the single person with the massive website is well
made - and moves forcibly against the idea of independence and my
early suggested aspects? And, in this context - I can see clearly
why you think the question of the independence/inter-dependence of
an evaluator from the site being evaluated is outside of the scope
of our charter.  But, I'm still not 100% convinced? 99% maybe ;-)

To my mind, the question is about trust - will the public place an
equal amount of trust in an evaluation done by a 1st party, as they
would a 3rd party?

Well, I think yes they could?

But, in our situation it might only be achieved under certain
circumstances?  Seemingly, a number of options exist:

1) Insist that the whole evaluation report be published, along with
the urls tested, procedure, etc...  The public could recreate your
tests and confirm your findings; or

2) Leave the publishing decision to the evaluation commissioner, and
instead set requirements for independence on the evaluator.

Note: In our methodology we already state "documentation need not
necessarily be public, as disclosure is up to the owner and/or
evaluation commissioner" - presumably you would not wish to publish
if you have assessed restricted areas of the website (which is a big
issue with option 1, amongst others)...

So? We already seem to be looking at the second option - hence the
reason I'm not yet 100% convinced that the question of evaluator
independence is currently entirely out of scope; or

3) Insist that some parts of every report are made public - not
urls, just procedures (as it should be the case that any similar
samples of web pages taken from the defined site will lead to pretty
much the same result)?

In light of your point, and the issues with option 1 further
investigation into option 3 might be an idea - it would of course
mean changes to 5a, and removing the idea of independence from the
scope.

Thoughts / comments welcome?
All the best

Alistair

On 21 Jun 2012, at 20:17, Peter Korn wrote:

Alistair,

If I am the only person involved in creating my own, fair sized
website (too large to feasibly evaluate every single page, being as
it is programmatically generated, etc. etc.), then
"self-assessment" means that I am also the assessor.  I cannot be
an ISO 9001:2000 compliant internal auditor.

Separate from that example, I don't understand why EvalTF should be
concerning itself with ISO 900x in any way.  Looking again at the
Objective portion of the Work Statement
<http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our
mandate is for a technical task (as I understand it): how to select
a representative sample of a site, how to aggregate results into an
overall conformance statement, etc.

The question of the independence/inter-dependence of an evaluator
from the site being evaluated is outside of the scope of our charter.


Peter


On 6/21/2012 11:06 AM, Alistair Garrison wrote:
Hi Peter,

Supporting 1st party assessment is as important to me as
supporting 3rd party assessment - which is why I based my proposal
on those well documented aspects you would look for in an internal
auditor for ISO 9001:2000.

Maybe, for clarity, it should have been 'not associated in their
day to day role with' - I think you have read 'associated' in the
same light as 'independent'.

Hope this helps.

Alistair

On 21 Jun 2012, at 19:36, Peter Korn wrote:

Alistair,

It wasn't clear to me that this was the outcome of our meeting.

Reviewing the EvalTF Work Statement
<http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence
of the Objective reads (*/emphasis added/*): "objective of Eval
TF is to develop an internationally harmonized methodology for
evaluating the conformance of websites to WCAG 2.0,/*that
supports different contexts, such as for self-assessment or
third-party evaluation*/ of small or larger websites".

If the methodology is to support self-assessment, then it cannot
define the evaluator as be different from the
developer/maintainer/accessibility-expert for the site.


Regards,

Peter

On 6/21/2012 10:02 AM, Alistair Garrison wrote:
Dear all,

In today's telecon, we discussed terms like independent when
talking about evaluations.  The outcome appeared to be that what
was needed was in fact a better definition for 'evaluator'.

I'm not going to propose the whole definition for 'evaluator',
however, just two aspects which we might consider including in the
definition:

Aspect 1) (of an evaluator) someone who is not responsible for the
accessibility of the website being evaluated.
Aspect 2) (of an evaluator) someone who is not associated with
developing and maintaining the website or its content.

Thoughts?

All the best

Alistair

--
<oracle_sig_logo.gif> <http://www.oracle.com/>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
<green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
Oracle is committed to developing practices and products that
help protect the environment




--
<oracle_sig_logo.gif> <http://www.oracle.com/>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
<green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
Oracle is committed to developing practices and products that help
protect the environment




--
<oracle_sig_logo.gif> <http://www.oracle.com/>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
<green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle
is committed to developing practices and products that help protect
the environment




--
Oracle <http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
developing practices and products that help protect the environment

This e-mail is confidential. If you are not the intended recipient you must not disclose or use the information contained within. If you have received it in error please return it to the sender via reply e-mail and delete any record of it from your system. The information contained within is not the opinion of Edith Cowan University in general and the University accepts no liability for the accuracy of the information provided.

CRICOS IPC 00279B
Received on Monday, 25 June 2012 00:37:06 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:14 GMT