RE: Review of EMMA usage in the Speech API (first editor's draft)

Sorry Glen,

I got busy with other things and need time to catch up on this tread.  Please hold off until the end of the week on the change.

Thanks


From: Glen Shires [mailto:gshires@google.com]
Sent: Tuesday, September 18, 2012 1:18 PM
To: Young, Milan; Jerry Carter; Satish S; public-speech-api@w3.org
Subject: Re: Review of EMMA usage in the Speech API (first editor's draft)

If there's no disagreement with the proposed text below (Sep 12) I will update the spec on Wednesday.
On Wed, Sep 12, 2012 at 8:56 AM, Glen Shires <gshires@google.com<mailto:gshires@google.com>> wrote:
There seems to be agreement on Jerry's wording except for the case that Satish raises in which conflicting data comes from both recognizers and can't be represented in EMMA (an EMMA attribute that can't be repeated with different values).  I propose the following slight modification to Jerry's wording that addresses this case. Here's my proposed full definition of the EMMA attribute. (The first two sentences are copied from the current definition in the spec.)

"EMMA 1.0 representation of this result. The contents of this result could vary across UAs and recognition engines, but all implementations must expose a valid XML document complete with EMMA namespace. UA implementations for recognizers that supply EMMA MUST contain all annotations and content generated by the recognition resources utilized for recognition, except where infeasible due to conflicting attributes.  The UA MAY add additional annotations to provide a richer result for the developer."

/Glen Shires

On Wed, Jun 20, 2012 at 10:03 AM, Young, Milan <Milan.Young@nuance.com<mailto:Milan.Young@nuance.com>> wrote:
Nice write-up.  I agree with the language.  I also agree that further details on running engines in parallel is necessary.


From: Jerry Carter [mailto:jerry@jerrycarter.org<mailto:jerry@jerrycarter.org>]
Sent: Tuesday, June 19, 2012 9:22 PM
To: Satish S; Young, Milan
Cc: public-speech-api@w3.org<mailto:public-speech-api@w3.org>

Subject: Re: Review of EMMA usage in the Speech API (first editor's draft)

Been off-line for a few days, so trying to catch up a bit...

Where we agree

It sounds like there is general agreement, but that additional word-smithing may be required.  By my reading of this thread, no one has objected to language allowing the UA to add additional annotations or to present a result which combines annotations from various sources.  Likewise, no one has objected to a goal of presenting the full set of annotations generated by the resources used.  This is encouraging common ground.


Single resource model: There might be no problem

The point of contention concerns how the UA handles cases where multiple resources might be used to build a result.  This may not be a real problem.  I do not see any language in the current draft which describes how multiple resources would be specified or acquired, so I'm not at all surprised that matters are unsettled.  One option is to draw the architectural lines such that there is never more than one recognition service.  Here the serviceURI would be required to point to a single logical entity rather than a document describing a collection of resources.  The single entity might employ disparate resources under the covers, but this would be outside of the Speech API specification.

In this case, the proposed language may be fine:

"The EMMA document MUST contain all annotations and content generated by the recognizer(s).  The UA MAY add additional annotations to provide a richer result for the developer."

and I can propose further language describing speech recognition services in more detail.


Dealing with Multiple Resources

Where the serviceURI  may refer to multiple resources, and at the risk of over generalizing, there are two cases.  The first case is where the 'start()' method invokes one or more resources and the result is built from all of them.  This is the most common case that I've seen in live systems.  There appears to be consensus that the result presented to the application author should contain the detailed annotations produced by the resources involved.

So we're left with the second case, where the 'start()' method invokes multiple resources but only a subset are used to generate the result.  Here at least one of the resources must be optional for result production, otherwise a recognition error would be generated.  Perhaps that optional resource was temporarily unavailable, or perhaps resource constraints prevented it from being used, or perhaps it failed to return a result before a timeout occurred.  Whatever the reason, the optional resource did not contribute to the recognition.  I have no problem with the EMMA result excluding any mention of the disregarded optional resource.  I also see nothing wrong with the UA adding annotations to describe any selection criteria used to generate the EMMA result.

Let me offer a slight variation on my original language and on Milan's proposal to capture this case:

"The EMMA document MUST contain all annotations and content generated by the recognition resources utilized for recognition.  The UA MAY add additional annotations to provide a richer result for the developer."

Again, further language describing speech recognition services will be helpful.


-=- Jerry



On Jun 19, 2012, at 11:13 AM, Satish S wrote:

Since there could be data coming from both recognizers and the UA might need to pick one of them and drop the rest (e.g. any EMMA attribute that can't be repeated with different values), we can't say the UA MUST send all content and all annotations. Using SHOULD allows the UA to implement this use case.

Cheers
Satish
On Tue, Jun 19, 2012 at 4:07 PM, Young, Milan <Milan.Young@nuance.com<mailto:Milan.Young@nuance.com>> wrote:
Good to hear.  So what did you think about the proposed text?  I didn't like the previous SHOULD-based suggestion.

Thanks


From: Satish S [mailto:satish@google.com<mailto:satish@google.com>]
Sent: Tuesday, June 19, 2012 4:27 AM
To: Young, Milan
Cc: Jerry Carter; Deborah Dahl; public-speech-api@w3.org<mailto:public-speech-api@w3.org>

Subject: Re: Review of EMMA usage in the Speech API (first editor's draft)


Yes I believe that was the use case Jerry mentioned earlier as well. There could be data coming from both recognizers and the UA might need to pick one of them (e.g. any EMMA attribute that can't be repeated with different values).

Cheers
Satish
On Tue, Jun 19, 2012 at 1:55 AM, Young, Milan <Milan.Young@nuance.com<mailto:Milan.Young@nuance.com>> wrote:
Satish, are you thinking of a scenario in which the UA runs multiple recognizers in parallel and then selects among the results?  If so, I think that's a reasonable use case, but I'd like to preserve Jerry's MUST  clause wrt annotations.  Could we agree on something like:

"The EMMA document MUST contain all annotations and content generated by the recognizer(s) that were used to produce the corresponding result.  The UA MAY add additional annotations to provide a richer result for the developer."

Thanks


From: Satish S [mailto:satish@google.com<mailto:satish@google.com>]
Sent: Friday, June 15, 2012 4:12 AM
To: Jerry Carter
Cc: Deborah Dahl; public-speech-api@w3.org<mailto:public-speech-api@w3.org>
Subject: Re: Review of EMMA usage in the Speech API (first editor's draft)

"The EMMA document MUST/SHOULD contain all annotations and content generated by the recognizer(s).  The UA MAY add additional annotations to provide a richer result for the developer."

If we use MUST above it would disallow UAs from selecting content from one recognizer over the other. So I think SHOULD would be more relevant.

Cheers
Satish
On Fri, Jun 15, 2012 at 12:06 PM, Jerry Carter <jerry@jerrycarter.org<mailto:jerry@jerrycarter.org>> wrote:

On Jun 15, 2012, at 5:06 AM, Satish S wrote:
Is this roughly what you had in mind?

I understood what Jerry wrote as
- There is a local recognizer, probably with a device specific grammar such as contacts and apps
- There is a remote recognizer that caters to a much wider scope
The UA would send audio to both and combine the results to deliver to Javascript

Jerry, could you clarify which use case you meant? The language I proposed was aimed towards a recognizer sitting outside the UA and generating EMMA data in which case it seemed appropriate that the UA would pass it through unmodified. If the UA is indeed generating EMMA data (whether combining from multiple recognizers or where the recognizer doesn't give EMMA data) it should be allowed to do so.

Your understanding is correct.  I've seen different architectures used.

Most often, there is a single vendor providing a single recognizer (either on device or in the cloud).   Also common, a single vendor will provide a consolidated product in which on device and in cloud resources are used together.  Here the interface may generate a single result which the UA would be free to pass along as-is.  In either case, the vendor will argue for using their result directly.

Then, there are rarer but real cases (e.g. certain Samsung products), where multiple venders are used in combination or as alternatives.  When used in combination, a consolidated recognition result would be generated outside what I think of as the recognition resource.  When used as alternatives, the results might be restructured or altered with the goal of providing consistent content or formats for developers.  Either way, some entity is preparing the final EMMA result.  Coming from the media resource side, I think of that entity as the UA.  Someone from the UA side might very well think of that as being just a different recognition resource!

In my mind and in the discussions as EMMA was coming to maturity, there is no reason that an EMMA result need pass through layers without modification.  There are, in fact, mechanisms within EMMA do describe intermediate results and their connections.  A speech recognition result might detail the phonetic (or sub-phonetic) lattice, the corresponding tokens from a grammar, and the semantic meaning as a derivation chain.  A hybrid resource might offer separate processing chains and then a unified result.  The example that drove much of the discussion was a map application with a touch screen.  The user says "How do I drive from here to here" with corresponding touches.  The EMMA result could include the entire recognition chain (phonetics -> tokens -> semantics) with the sequence of touches (touch1, then touch2) and then produce a final result (mode=drive + location1 + location2) for passing to a routing application.

What is critical is that the application developer have access to the information required for their task.  Developers, in my experience, object when information is stripped.  Media resource vendors, working in response to developer requests, want assurances that additional details that they add will be passed through to the developer.  The current language

UA implementations for recognizers that supply EMMA must pass that EMMA structure directly.

is too restrictive.   Let me suggest instead

"The EMMA document MUST/SHOULD contain all annotations and content generated by the recognizer(s).  The UA MAY add additional annotations to provide a richer result for the developer."

I offered SHOULD or MUST.  I prefer MUST because I believe that the contents of a result generated by the recognizer exists for a reason.  I can accept SHOULD if there is a strong argument for presenting a simplified or altered result.

Milan and Salish, could you elaborate on what you had in mind when you raised concerns about the UA modifying the speech recognizer's EMMA?

The primary reason I added that clause in was to preserve those EMMA attributes (emma:process, ...) from the recognizer to JS without calling out specific attributes. Since we agreed that instead of calling out attributes we'll add use cases as examples, there is lesser reason for this clause now and I agree it does enable use cases like what I mentioned above. So I'm fine dropping that clause if there are no other strong reasons to keep it in.

--
Cheers
Satish

Makes sense.

-=- Jerry