RE: Another comment about confidence value.

Ok I've sat back and read most people's thoughts on this subject and
would now like to ask a question of those who believe we should include
a confidence level. I personally still think this is a bad idea for all
the same reasons I stated in my original email. I feel priority and/or
severity levels are the most widely used and understood mandatory fields
in a defect tracking tool and even then they are almost always misused
at least once on any given project when working with external parties
outside of your control - especially by 'developers' who think they have
the aptitude of a test analyst, but do not. Introducing a confidence
level will simply make defect report writing and evaluation more time
consuming. You can argue until the pigs come home, but we will not use a
confidence level in our reporting. 
 
Q:
For companies who use disabled users; how do you suggest they measure
the confidence level of their output? (I'm not assuming you can cover
every disability across every project or even one project - but it's
compulsory in my opinion to 'try', using the usual quality triangle to
ensure testing is cost/time effective).
 
Situation: Dyslexic user is provided with high-level test case
scenarios, where auditors drill down further with detailed documented
test scripts using both manual and automated methods. The dyslexic user
has a problem with the complexity of the copy in two areas of the
website. This type of defect is not picked up by the auditor or the
tool, nor is it appreciated by the auditor. How do you measure the
confidence level of those two defects?
 
We have some of the most highly skilled and experienced test analysts
and developers who have worked for companies such as AOL since 1994 and
were responsible for the entire test management and execution and
International beta coordination of all new client software and
technology for the UK and Sweden whilst providing ongoing support to
Germany and France - trust me when I say they are more experienced than
most when it comes to 'testing' Internet technologies.
 
I am not saying for one second that what I'm saying is gospel; I'm
simply providing you with some background to the people who have this
opinion.
 
We use both manual and automated testing methods where the former
outweighs the latter by a long way. If someone is less than certain
about the output of their test they will always seek a second opinion
from their colleagues. This is why it's absolutely necessary to have a
team of auditors on any project. Each person's interpretation of an
outcome is debated until they come to an agreement. The combined
interpretation may not be 100% accurate if compared to that of a
disabled user (or even someone outside the company), but at least they
are 100% confident in the recorded defect.  Anything less than this is
not good enough.
 
This type of testing is not scientific so stop trying to make it so.
 
Kind regards,
Paul
p.s. please excuse any typos etc., I'm feeling a little tired and unable
to verify my own writing right now.
 
-----Original Message-----
From: public-wai-ert-request@w3.org
[mailto:public-wai-ert-request@w3.org] On Behalf Of Nils Ulltveit-Moe
Sent: 18 April 2005 20:08
To: Charles McCathieNevile
Cc: Giorgio Brajnik; public-wai-ert@w3.org
Subject: Re: Another comment about confidence value.
 
 
Hi Charles,
 
I would suggest that confidence is interpreted as it is defined in
statistics; i.e. as a probability which is a real number from and
inclusive 0 and to and inclusive 1. 
 
If some applications want to map this basic definition to percentage or
some other scheme, then they can define their own mappings, e.g. "low is
confidence less than 0.3, medium is confidence between 0.3 and 0.7 and
high is confidence greater than 0.7". or "Confidence percentage is
probability times 100". Those implementors that need such mappings can
provide customized RDF/OWL schema and constraints to give a unique
interpretation for their specific applications. 
 
Using the probability is a generally accepted way of representing
confidence values in most areas of science. Other representations can be
derived from this.
 
Mvh.
Nils
 
man, 18,.04.2005 kl. 19.47 +0200, skrev Charles McCathieNevile:
> For example, if Chris uses "high, medium, low" and Giorgio uses a
number  
>  from 1 to 7 and Nils uses an integer from 0 to 100, I can map Chris'

> confidence to 1, 4 and 7 on Giorgio's scale and map that to some
numbers  
>  from Nils. Equally, I can decide to do a little more work and map
some of  
> Chris' results to 1, 4 and 7, some of them to 3, 5, 7 and some to 1,
3, 4  
> according to what the test is...
> 
> Does this sound like what others are thinking?
> 
> cheers
> 
> Chaals
> 
-- 
Nils Ulltveit-Moe <nils@u-moe.no>