W3C home > Mailing lists > Public > public-speech-api@w3.org > June 2012

Re: Confidence property

From: Glen Shires <gshires@google.com>
Date: Fri, 15 Jun 2012 12:00:04 -0700
Message-ID: <CAEE5bcj+ntmtrU5MhOYm=CP3igmAFF+1vjUVmaayu+Mb_MSt7Q@mail.gmail.com>
To: "Young, Milan" <Milan.Young@nuance.com>
Cc: public-speech-api@w3.org
It may be that we have a misunderstanding in how we both define "native
confidence values".  I have been using that term, and continue to use that
term to indicate a 0.0 - 1.0 scale that has not had any skew applied to
make 0.5 reasonable.  I have not been using that term to refer to any
internal recognizer scale that is other than 0.0 - 1.0.

Comments inline below...

On Thu, Jun 14, 2012 at 6:04 PM, Young, Milan <Milan.Young@nuance.com>
 wrote:

> You argue that there exists some recognizer that is NOT capable of giving
> a meaningful native interpretation to thresholds like Ď0.5í.  I will accept
> that.
>
[Glen] Thank you

> ****
>
> ** **
>
> You further suggest that these same recognizer(s) have some magic ability
> to transform these thresholds to something that IS meaningful.  I will
> accept that too.  Letís call that magic transformation webToInternal() and
> itís inverse internalToWeb().
>
 [Glen] OK

> ****
>
> ** **
>
> Without requiring this engine to expose internalToWeb() a developer could
> set a threshold like ď0.5Ē and get back score like ď0.1Ē.  If you were a
> developer, would that make sense to you?
>
[Glen] Yes

  What practical use would you even have for such a number?
>
[Glen] I believe most Group 2 web developers don't care to look at
confidence values:

 - Some will simply set nomatchThreshold = 0.5 and control their
application based on whether onresult or onnomatch fires.

 - Some more sophisticate Group 2 developers will set nomatchThreshold =
0.5 and may increment it up or down based on if onresult or onnomatch is
firing too often or rarely.

 - Only the most sophisticated Group 2 developers will look at the
confidence values returned in the results or in emma. Since they are
processing them in a recognition-dependent manner, they must only compare
relative values. For example, if they find that the second alternative has
a confidence value relatively near the first, the app may ask the user to
disambiguate.  Using the example you give, if the top result is 0.1 and the
second result is 0.085, the app could ask the user to disambiguate.

For Group 3 developers that do process these values, getting back the 0.1
result is invaluable, because it matches the native levels in their tuning
tools, logs and other applications.

So yes, this has very practical uses and benefits for Group 2 and Group 3
developers.


> It may as well be a Chinese character.
>
[Glen] Fortunately, it is a float, and can easily be compared against other
float values.

> ****
>
> ** **
>
> Wouldnít it be a lot more useful to developers and consistent with
> mainstream engines to simply require support for internalToWeb()?  Iím sure
> folks that are capable of building something as complicated as a recognizer
> can solve an math equation.  Iíll even offer to include my phone number in
> the spec so that they can call me for help J.
>
[Glen] No. This would be very problematic for Group 3 developers that use
these recognizers. Their tuning tools, their logs, their other applications
all may be based on native confidence values, and this complicates their
implementation, as you have pointed out. Instead, Group 3 developers would
much prefer to only use native values, which they can do because the native
values are returned in the results and in emma. Yes, they do have to
copy-and-paste a short JavaScript function for this, but that's trivial.
 For Group 2 and Group 1 developers, there's no difference whether these
recognizers support internalToWeb().

Thanks
>
Thank you
Received on Friday, 15 June 2012 19:01:19 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Friday, 15 June 2012 19:01:22 GMT