The HTML Speech protocol is defined as a sub-protocol of WebSockets [WS-PROTOCOL], and enables HTML user agents and applications to make interoperable use of network-based speech service providers, such that applications can use the service providers of their choice, regardless of the particular user agent the application is running in. The protocol bears some similarity to [MRCPv2]. However, since the use cases for HTML Speech applications are in some places considerably different from those around which MRCPv2 was designed, the HTML Speech protocol is not merely a transcript of MRCP, but shares some design concepts, while simplifying some details, and adding others. Similarly, because the HTML Speech protocol builds on WebSockets, its session negotiation and media transport needs are quite different from those of MRCP.
TODO: Michael Johnston will add a sentence or two about the higher level motivation.
This document is an informal rough draft that collates proposals, agreements, and open issues on the design of the necessary underlying protocol for the HTML Speech XG, for the purposes of review and discussion within the XG.
Client
|-----------------------------|
| HTML Application | Server
|-----------------------------| |--------------------------|
| HTML Speech API | | Synthesizer | Recognizer |
|-----------------------------| |--------------------------|
| HTML-Speech Protocol Client |---html-speech/1.0 subprotocol---| HTML-Speech Server |
|-----------------------------| |--------------------------|
| WebSockets Client |-------WebSockets protocol-------| WebSockets Server |
|-----------------------------| |--------------------------|
A Recognizer performs speech recognition, with the following characteristics:
Because continuous recognition plays an important role in HTML Speech scenarios, a Recognizer is a resource that essentially acts as a filter on its input streams. Its grammars/language models can be specified and changed at any time, as needed by the application, and the recognizer adapts its processing accordingly. Single-shot recognition (e.g. a user on a web search page presses a button and utters a single web-search query) is a special case of this general pattern, where the application specifies its model once, and is only interested in one match event, after which it stops sending audio (if it hasn't already).
"Recognizers" are not strictly required to perform speech recognition, and may perform additional or alternative functions, such as speaker verification, emotion detection, or audio recording.
A Synthesizer generates audio streams from textual input, with the following characteristics:
Because playback is done asynchronously by the user agent if and when the application deems appropriate, a Synthesizer resource does not provide any form of shuttle control (pausing or skipping) or volume control. Nor does it have any concept of queuing. It simply services each synthesis request it receives, rendering the audio at least as rapidly as would be needed to support real-time playback, and preferably faster. The client may make multiple simultaneous requests, which the server must service simultaneously, and for which it may stream the rendered audio simultaneously.
TODO: There were some clarifying questions around this in the spec review. Robert Brown to expand on this, perhaps with an example.
The WebSockets session is established through the standard WebSockets HTTP handshake, with these specifics:
TODO: Clarify that service parameters may be specified in the query string, but may be overridden using messages in the html-speech/1.0 websockets protocol once the websockets session has been established.
For example:
C->S: GET /speechservice123?customparam=foo&otherparam=bar HTTP/1.1
Host: examplespeechservice.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: OIUSDGY67SDGLjkSD&g2 (for example)
Sec-WebSocket-Version: 9
Sec-WebSocket-Protocol: html-speech/1.0, x-proprietary-speech
S->C: HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: html-speech/1.0
Once the WebSockets session is established, the UA can begin sending requests and media to the service, which can respond with events, responses or media, in the case of TTS.
NOTE: In MRCP, session negotiation also involves negotiating unique channel IDs (e.g. 128397521@recognizer) for the various resource types the client will need (recognizer, synthesizer, etc). In html-speech/1.0 this is unnecessary, since the WebSockets connection itself provides a unique shared context between the client and server, and resources are referred to directly by type, without the need for channel-IDs.
TODO: clarify that advanced scenarios involving multiple engines of the same resource type, or using the same input audio stream for consumption by different types of vendor-specific resources, are out of scope. These may be implemented behind the scenes by the service.
The signaling design borrows its basic pattern from [MRCPv2], where there are three classes of control messages:
The interaction is full-duplex and asymmetrical: service activity is instigated by requests the UA, which may be multiple and overlapping, where each request results in one or more messages from the service back to the UA.
control-message = start-line ; i.e. use the typical MIME message format
*(header CRLF)
CRLF
[body]
start-line = request-line | status-line | event-line
header = <Standard MIME header format> ; actual headers depend on the type of message
body = *OCTET ; depends on the type of message
generic-header =
| accept
| accept-charset
| content-base
| logging-tag
| resource-id
| vendor-specific
| content-type
| content-encoding
resource-id = "Resource-ID:" ("recognizer""listener" | "synthesizer" | vendor-resource)
vendor-resource = "x-" 1*UTFCHAR
accept = <same as [MRCPv2]>
accept-charset = <same as [MRCPv2]>
content-base = <same as [MRCPv2]>
content-type = <same as [MRCPv2]>
content-encoding = <same as [MRCPv2]>
logging-tag = <same as [MRCPv2]>
vendor-specific = <same as [MRCPv2]>
NOTE: This is mostly a strict subset of the [MRCPv2] generic headers, many of which have been omitted as either unnecessary or inappropriate for HTML speech client/server scenarios.
Request messages are sent from the client to the server, usually to request an action or modify a setting. Each request has its own request-id, which is unique within a given WebSockets html-speech session. Any status or event messages related to a request use the same request-id. All request-ids MUST have an integer value between 0 and 65535.
NOTE: This upper value is less that that used in MRCP, and was chosen so that request-ids can be associated with audio streams using a 16-bit field in binary audio messages.
TODO: revise this number. 2^16-1 seems small. It may have security implications. Perhaps 2^24-1?
request-line = version SP message-length SP method-name SP request-id SP CRLF
version = "html-speech/" 1*DIGIT "." 1*DIGIT ; html-speech/1.0
message-length = 1*DIGIT
method-name = general-method | synth-method | reco-method | proprietary-method
request-id = <Decimal integer between 0 and 65535>
TODO: Add some examples.
Status messages are sent by the server, to indicate the state of a request.
status-line = version SP message-length SP request-id SP status-code SP request-state CRLF
status-code = 3DIGIT ; Specific codes TBD, but probably similar to those used in MRCP
; All communication from the server is labeled with a request state.
request-state = "COMPLETE" ; Processing of the request has completed.
| "IN-PROGRESS" ; The request is being fulfilled.
| "PENDING" ; Processing of the request has not begun.
Event messages are sent by the server, to indicate specific data, such as synthesis marks, speech detection, and recognition results. They are essentially specialized status messages.
event-line = version SP message-length SP event-name SP request-id SP request-state CRLF
event-name = synth-event | reco-event | proprietary-event
HTML Speech applications feature a wide variety of audio transmission scenarios. The number of audio streams at any given time is not fixed. A recognizer may accept one or more input streams, which may start and end at any time as microphones or other input devices are activated/deactivated by the application or the user. Recognizers do not require their data in real-time, and will generally prefer to wait for delayed packets where a human listener would rather just skip the data and put it down to a "bad line". Applications may, and often will, request the synthesis of multiple SSML documents at the same time, for playback at the application's discretion. The synthesizer needs to return rendered data to the client rapidly (generally faster than real time), and MAY render multiple requests in parallel if it has the capacity to do so.
In html-speech/1.0, audio (or other media) is packetized and transmitted as a series of WebSockets binary messages, on the same WebSockets session used for the control messages.
audio-packet = binary-message-type
binary-request-id
binary-reserved
binary-data
binary-message-type = OCTET ; Values > 0x03 are reserved. 0x00 is undefined.
binary-session-id = 2OCTET ; Matches the request-id for a SPEAK or START-MEDIA-STREAM request
binary-reserved = OCTET ; Just buffers out to the 32-bit boundary for now, but may be useful later.
binary-data = *OCTET
TODO: reduce message-type to a few bits, and expand request-id.
The binary-request-id field is used to correlate the audio stream with the corresponding SPEAK or START-MEDIA-STREAM request. It is a 16-bit unsigned integer.
The binary-message-type field has these defined values:
TODO: Change "Audio" to "Media". Clarify that its encoding is specified by a mime content type in the request that initiated the stream (SPEAK or START-MEDIA-STREAM), and while it will usually be some form of audio encoding, it MAY be any content type, including text, pen strokes, touches/clicks, compass bearings, etc.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 0 0 0 0 0 0 0|1 0 0 0 1 1 1 1 0 0 0 0 0 1 0 1|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+
| encoded audio data |
| ... |
| ... |
| ... |
+---------------------------------------------------------------+
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|0 1 0 0 0 0 0 0|1 0 0 0 1 1 1 1 0 0 0 0 0 1 0 1|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+
| timestamp |
|1 0 1 1 1 0 0 1 1 1 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0|
+---------------------------------------------------------------+
TODO: DELETE the Skip message type.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 1 0 0 0 0 0 0|1 0 0 0 1 1 1 1 0 0 0 0 0 1 0 1|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+
TODO: add a paragraph about the interleaving of audio and signaling.
There is no strict constraint on the size and frequency of audio messages. Nor is there a requirement for all audio packets to encode the same duration of sound. Implementations SHOULD seek to minimize interference with the flow of other messages, by sending messages that encode between 20 and 80 milliseconds of media. Since a WebSockets frame header is typically only 4 bytes, overhead is minimal and implementations should err on the side of sending smaller packets more frequently.
A sequence of audio messages represents an in-order contiguous stream of audio data. Because the messages are sent in-order and audio packets cannot be lost (WebSockets uses TCP), there is no need for sequence numbering or timestamps. The sender just packetizes audio from the encoder and sends it, while the receiver just un-packs the messages and feeds them to the decoder. Timing is calculated by decoded offset from the beginning of the stream.
A synthesis service MAY (and typically will) send audio faster than real-time, and the client MUST be able to handle this.
A client MAY send audio to a recognition service slower than real time. While this generally causes undesirable user interface latency, it may be necessary due to practical limitations of the network. A recognition service MUST be prepared to receive slower-than-real-time audio.
Although most services will be strictly either recognition or synthesis services, some services may support both in the same session. While this is a more advanced scenario, the design does not introduce any constraints to prevent it. Indeed, both the client and server MAY send audio streams in the same session.
Advanced implementations of HTML Speech may incorporate multiple channels of audio in a single transmission. For example, living-room devices with microphone arrays may send separate streams in order to capture the speech of multiple individuals within the room. Or, for example, some devices may send parallel streams with alternative encodings that may not be human-consumable (like standard codecs) but contain information that is of particular value to a recognition service. The protocol future-proofs for this scenario by incorporating a channel ID into each message, so that separate audio channels can be multiplexed onto the same session. Channel IDs are selected by the originator of the stream, and only need to be unique within the set of channels being transmitted by that originator.
TODO: add a C->S message for sending metadata to the recognizer.
TODO: Michael Johnston to write the rational for why we mix things.
TODO: note that at least muLaw/aLaw/PCM must be supported.
TODO: There is an open issue to do with transitive access control. The client sends a URI to the service, which the client can access, but the service cannot, because it is not authorized to do so. How does the client grant access to the resource to the service? There are two design contenders. The first is to use the cookie technique that MRCP uses. The second is to use a virtual tag, which we discussed briefly at the F2F - Michael Bodell owes a write-up. In the absence of that write-up, perhaps the default position should be to use cookies.
TODO: Specify which headers are sticky. URI request parameters aren't standardized. The standard way to set parameters is SET-PARAMS, and this does so globally. Individual requests may set different values, but these are local to that request.
The GET-PARAMS and SET-PARAMS requests are the same as their [MRCPv2] counterparts. They are used to discover and set the configuration parameters of a resource (recognizer or synthesiser). Like all messages, they must always include the Resource-ID header.
general-method = "SET-PARAMS"
| "GET-PARAMS"
capability-query-headers =
"Supported-Media:" mime-media-type *("," mime-media-type) ; See [RFC3555]
| "Supported-Languages:" lang-tag *("," lang-tag) ; See [RFC5646]
Two additional headers, "Supported-Media" and "Supported-Languages" are introduced in html-speech/1.0 to provide a way for the application/UA to determine whether a resource supports the basic language and codec capabilities it needs. In most cases applications will know service's resource capabilities ahead of time. However, some applications may be more adaptable, or may wish to double-check at runtime. To determine resource capabilities, the UA sends a GET-PARAMS request to the resource, containing a set of capabilities, to which the resource responds with the specific subset it actually supports.
TODO: how do we check for other configuration settings? e.g. what grammars are available? e.g. supported grammar format (srgs-xml vs srgs-ebnf vs some-slm-format).
ISSUE: this could become unwieldy as more parameters are added. Is there a more generic approach?
For example:
C->S: html-speech/1.0 GET-PARAMS 34132
resource-id: recognizer
supported-media: audio/basic, audio/amr-wb,
audio/x-wav;channels=2;formattag=pcm;samplespersec=44100,
audio/dsr-es202212; rate:8000; maxptime:40
supported-languages: en-AU, en-GB, en-US, en (The application has UK, US and Australian English speakers, and possibly speakers of other English dialects)
S->C: html-speech/1.0 34132 200 COMPLETE
resource-id: recognizer
supported-media: audio/basic, audio/dsr-es202212; rate:8000; maxptime:40
supported-languages: en-GB, en (The recognizer supports UK English, but will work with any English)
C->S: html-speech/1.0 GET-PARAMS 48223
resource-id: synthesizer
supported-media: audio/ogg, audio/flac, audio/basic
supported-languages: en-AU, en-GB
S->C: html-speech/1.0 48223 200 COMPLETE
resource-id: synthesizer
supported-media: audio/flac, audio/basic
supported-languages: en-GB
Speech services may care to send optional vendor-specific interim events during the processing of a request. For example: some recognizers are capable of providing additional information as they process input audio; and some synthesizers are capable of firing progress events on word, phoneme, and viseme boundaries. These are exposed through the HTML Speech API as events that the webapp can listen for if it knows to do so. A service vendor MAY require a vendor-specific value to be set with SET-PARAMS before a it starts to fire certain events.
interim-event = version request-ID SP INTERIM-EVENT CRLF
*(header CRLF)
CRLF
[body]
The Request-ID and Content-Type headers are required, and any data conveyed by the event must be contained in the body.
TODO: in an implementation report, would just need to demonstrate some usage, not all usages of this, just to demostrate that the plumbing works.
Generally, a service may only send a message to the client in response to a client-originating request. However, in some cases, a service-originating notification mechanism is useful.
server-notification = version SP NOTIFY CRLF
*(header CRLF)
CRLF
[body]
The NOTIFY message is sent from the service to the client. It has no Request-ID, and there is no response from the client to the service. It MAY include a Resource-ID if relevant. It MUST include a Content-Type header if it contains a body. It MAY include Vendor-Specific-Parameters. Other headers are not supported, since it is unclear hwo the client would make sense of them. If webapps require support for request-ids, parameter, etc, they would probably be best encoded within the message body.
If the [body] was detected as being XML or JSON, it would be nice if the client browser could automatically reflect the data as a DOM or EMCA object.
ISSUE: Is there a clear scenario for requestless notifications? What does it mean for unsolicited information to just pop out of the API into the app? If so, we need to be more specific about what the scenario is, and how the NOTIFY message is used. If not, we should delete this message from the draft.
A recognizer resource is either in the "listening" state, or the "idle" state. Because continuous recognition scenarios often don't have dialog turns or other down-time, most functions can be performed in either state: grammars can be loaded, rules activated and deactivated, media streams started, and text-input interpreted, regardless of whether the recognizer is listening or idle. For example: text dictation applications commonly have a variety of command grammars that are activated and deactivated to enable editing and correction modes; in open-microphone multimodal applications, the application will listen continuously, but change the set of active grammars based on the user's other non-speech interactions with the app. The key distinction between the idle and listening states is the obvious one: when listening, the recognizer processes incoming media and produces results; whereas when idle, the recognizer SHOULD buffer audio but will not process it.
TODO: some turn based recognizers can't change state in a recognition. What happens then?
TODO: what happens to timers if we enter the listening state without having an input stream? Timers should be based on the start of the input stream.
Recognition is accomplished with a set of messages and events, some of which are borrowed from [MRCPv2] and have the same or similar semantics, while others are unique to html-speech/2.0 and are needed to support HTML Speech scenarios.
ISSUE: If we're going so far as to replace RECOGNIZE with LISTEN + listen-mode, then why not replace the "recognizer" resource with a "listener", and call the whole topic "listening" rather than "recognition"? NOTE: this was resolved last week as "listener", but upon rediscussion this week, it's less controversial to just call it "recognizer".
Idle State Listening State
| |
|--\ |--\
| START-MEDIA-STREAM | START-MEDIA-STREAM
|<-/ |<-/
| |
|--\ |--\
| DEFINE-GRAMMAR | DEFINE-GRAMMAR
|<-/ |<-/
| |
|--\ |--\
| SET-GRAMMAR | SET-GRAMMAR
|<-/ |<-/
| |
|--\ |--\
| INTERPRET | INTERPRET
|<-/ |<-/
| |
|--\ |--\
| INTERPRETATION-COMPLETE | INTERPRETATION-COMPLETE
|<-/ |<-/
| |
|--\ |--\
| STOP | LISTEN
|<-/ |<-/
| |
| |--\
| | INTERIM-EVENT
| |<-/
| |
| |--\
| | START-OF-INPUT
| |<-/
| |
| |--\
| | START-INPUT-TIMERS
| |<-/
| |
| |--\
| | END-OF-INPUT
| |<-/
| |
| |--\
| | INTERMEDIATE-RESULT
| |<-/
| |
| |--\
| | RECOGNITION-COMPLETE
| | (when mode = recognize-continuous)
| |<-/
| |
|<---RECOGNITION-COMPLETE----|
|(when mode = recognize-once)|
| |
|<----------STOP-------------|
| |
|<---some 4xx/5xx errors-----|
reco-method = "CLEAR-GRAMMARS" ; Unloads all grammars, whether active or inactive
| "DEFINE-GRAMMAR" ; Pre-loads & compiles a grammar, assigns a temporary URI for reference in other methods
| "INTERPRET" ; Interprets input text as though it was spoken
| "LISTEN" ; Transitions Idle -> Listening
| "SET-GRAMMAR" ; Activates and deactivates grammars and rules
| "START-INPUT-TIMERS" ; Starts the timer for the various input timeout conditions
| "START-MEDIA-STREAM" ; Starts an input media stream for the recognizer to listen to
| "STOP" ; Transitions Listening -> Idle
In continuous recognition, a variety of grammars may be loaded over time, potentially resulting in unused grammars consuming memory resources in the recognizer. The CLEAR-GRAMMARS method unloads all grammars, whether active or inactive.
The DEFINE-GRAMMAR method is similar to its namesake in [MRCPv2]. DEFINE-GRAMMAR does not activate a grammar, it simply causes the recognizer to pre-load and compile it, and associates it with a temporary URI that can then be used to activate or deactivate the grammar or one of its rules. DEFINE-GRAMMAR is not required in order to use a grammar, since the recognizer can load grammars as needed. However, it is useful when an application wants to ensure a large grammar is pre-loaded and ready for use prior to the recognizer entering the listening state. DEFINE-GRAMMAR can be used when the recognizer is in either the listening or idle state.
The INTERPRET method is similar to its namesake in [MRCPv2], and processes the input text according to the set of grammar rules that are active at the time it is received by the recognizer. It MUST include the Interpret-Text header. The use of INTERPRET is orthogonal to any audio processing the recognizer may be doing, and will not affect any audio processing. The recognizer can be in either the listening or idle state.
The LISTEN method transitions the recognizer from the idle state to the listening state. The recognizer then processes the media input streams against the set of active grammars. The request MUST include the Source-Time header, which is used by the Recognizer to determine the point in the input stream(s) that the recognizer should start processing from. The request MUST also include the Listen-Mode header to indicate whether the recognizer should perform continuous recognition, a single recognition, or vendor-specific processing.
A LISTEN request MAY also activiate or deactivate grammars and rules using the Grammar-Activate and Grammar-Deactivate headers. These grammars/rules are considered to be activated/deactivated from the point specified in the Source-Time header.
NOTE: LISTEN does not use the same grammar specification technique as the MRCP RECOGNIZE method. In html-speech/1.0 this would add unnecessary and redundant complexity, since all the necessary functionality is already present in other html-speech/1.0 methods.
If or when there are no input media streams, the recognizer automatically transitions to the idle state, and issues a RECOGNITION-COMPLETE event, with Completion-Cause set to ???TBD.
TODO: should be "when" not "if".
TODO: Specify Completion-Cause value for no input stream.
If the recognizer is already in the listening state when it receives a LISTEN request, it remains in the listening state, changes the Listen-Mode to match that of the new request, and activates/deactivates any grammars/rules specified in the new request. Any grammars/rules active prior to the request, but not explicitly deactivated by the request, remain active.
The SET-GRAMMAR method is used to activate and deactivate grammars and rules, using the Grammar-Activate and Grammar-Deactivate headers. The Source-Time header MUST be used, and activations/deactivations are considered to take place at precisely that time in the input stream(s).
ISSUE: Do we need an explicit method for this, or is SET-PARAMS enough? One option is to not allow them on set/get-params. Another is to say that if get/set-params does exactly the same thing, then there's no need for this method. If there's a default set of active grammars, then get-params might be required. Get-params may also be useful for defensive programming. Inline grammars don't have URIs. Suggestion is to add a GET-GRAMMARS, and disallow get/set-params.
This is identical to the [MRCPv2] method with the same name. It is useful, for example, when the application wants to enable voice barge-in during a prompt, but doesn't want to start the time-out clock until after the prompt has completed.
The START-MEDIA-STREAM request defines and initiates an audio input stream. It MUST include the Source-Time header, which the recognizer will use as the start-time for the input stream, so that other requests and events can refer to precise times in the input stream. It MUST also include the Audio-Codec header, which specifies the codec and parameters used to encode the media stream. The Request-ID for a START-MEDIA-STREAM request is used in the associated audio messages. The recognizer should respond to the START-MEDIA-STREAM request with a 200 IN-PROGRESS status message if it is able to accept the stream. When the stream ends, the recognizer should send a 200 COMPLETE status message.
TODO: collapse this with LISTEN.
Recognition events are associated with 'IN-PROGRESS' request-state notifications from the 'recognizer' resource.
reco-event = "END-OF-INPUT" ; End of speech has been detected
| "INTERIM-EVENT" ; See Interim Events above
| "INTERMEDIATE-RESULT" ; A partial hypothesis
| "INTERPRETATION-COMPLETE"
| "RECOGNITION-COMPLETE" ; Similar to MRCP2 except that application/emma+xml (EMMA) will be the default Content-Type.
| "START-OF-INPUT" ; Start of speech has been detected
ISSUE: "RECOGNITION-COMPLETE" seems like an inappropriate name, since in continuous recognition, it isn't complete. "RECOGNITION-RESULT" would be a better name.
TODO: change to START/END-OF-SPEECH
END-OF-INPUT is the logical counterpart to START-OF-INPUT, and indicates that speech has ended. The event MUST include the Source-Time header, which corresponds to the point in the input stream where the recognizer estimates speech to have ended, NOT when the endpointer finally decided that speech ended, which is a number of milliseconds later.
See Interim Events above.
Continuous speech (aka dictation) often requires feedback about what has been recognized thus far. Waiting for a RECOGNITION-COMPLETE event prevents this sort of user interface. INTERMEDIATE-RESULT provides this intermediate feedback. As with RECOGNITION-COMPLETE, contents are assumed to be EMMA unless an alternate Content-Type is provided.
This event is identical to the [MRCPv2] event with the same name.
This method is similar to the [MRCPv2] method with the same name, except that application/emma+xml (EMMA) is the default Content-Type. The Source-Time header must be included, to indicate the point in the input stream when the event occured. When the Listen-Mode is reco-once, the recognizer will transition from the listening state to the idle state when this message is fired, and the Recognizer-State header in the event is set to "idle".
TODO: Describe how final results can be replaced in continuous recognition.
TODO: no match is returned, is the EMMA no-match document required?
Indicates that start of speech has been detected. The Source-Time header MUST correspond to the point in the input stream(s) where speech was estimated to begin, NOT when the endpointer finally decided that speech began (a number of milliseconds later).
The list of valid headers for the recognizer resource include a subset of the [MRCPv2] Recognizer Header Fields, where they make sense for HTML Speech requirements, as well as a handful of headers that are required for HTML Speech.
reco-header = Confidence-Threshold
| Sensitivity-Level
| Speed-Vs-Accuracy
| N-Best-List-Length
| No-Input-Timeout
| Recognition-Timeout
| Waveform-URI
| Media-Type
| Input-Waveform-URI
| Completion-Cause
| Completion-Reason
| Recognizer-Context-Block
| Start-Input-Timers
| Speech-Complete-Timeout
| Speech-Incomplete-Timeout
| Failed-URI
| Failed-URI-Cause
| Save-Waveform
| Speech-Language
| Hotword-Min-Duration
| Hotword-Max-Duration
| Interpret-Text
| audio-codec ; The audio codec used in an input media stream
| grammar-activate ; Specifies a grammar or specific rule to activate.
| grammar-deactivate ; Specifies a grammar or specific rule to deactivate.
| hotword ; Whether to listen in "hotword" mode (i.e. ignore out-of-grammar speech)
| listen-mode ; Whether to do continuous or one-shot recognition
| partial ; Whether to send partial results
| partial-interval ; Suggested interval between partial results, in milliseconds.
| recognizer-state ; Indicates whether the recognizer is listening or idle
| source-time ; The UA local time at the request was initiated
| user-id ; Unique identifier for the user, so that adaptation can be used to improve accuracy.
hotword = "Hotword:" BOOLEAN
listen-mode = "Listen-Mode:" ("reco-once" | "reco-continuous" | vendor-listen-mode)
vendor-listen-mode = "x-" 1*UTFCHAR
recognizer-state = "Recognizer-State:" ("listening" | "idle")
source-time = "Source-Time:" 1*19DIGIT
audio-codec = "Audio-Codec:" mime-media-type ; see [RFC3555]
partial = "Partial:" BOOLEAN
partial-interval = "Partial-Interval:" 1*5DIGIT
grammar-activate = "Grammar-Activate:" "<" URI ["#" rule-name] [SP weight] ">" *("," "<" URI ["#" rule-name] [SP weight] ">")
rule-name = 1*UTFCHAR
weight = "0." 1*3DIGIT
grammar-deactivate = "Grammar-Deactivate:" "<" URI ["#" rule-name] ">" *("," "<" URI ["#" rule-name] ">")
user-id = "User-ID:" 1*UTFCHAR
TODO: discuss how recognition from file would work.
ISSUE: Does this imply further API requirements? For example, START-INPUT-TIMERS is there for a good reason, but AFAIK we haven't spoken about it. Similarly, Early-No-Match seems useful to developers. Is it?
Headers with the same names as their [MRCPv2] counterparts are considered to have the same specification. Other headers are describe as follows:
The Audio-Codec header is used in the START-MEDIA-STREAM request, to specify the codec and parameters used to encode the input stream, using the MIME media type encoding scheme specified in [RFC3555].
The Grammar-Activate header specifies a list of grammars, and optionally specific rules within those grammars, to be activated. If no rule is specified for a grammar, the root rule is activated. It may also specify the weight of the rule. The Grammar-Activate header MAY be used in both the SET-GRAMMAR and LISTEN methods.
ISSUE: Including a header multiple times, rather than trying to encode a list with delimiters, works great for SIP's Record-Route header. I don't know if there are any gotchas with this approach.
ISSUE: Grammar-Activate/Deactivate probably don't make sense in GET/SET-PARAMS. Is this an issue? Perhaps this would be better achieved in the message body? The same format could be used.
The Grammar-Deactivate header specifies a list of grammars, and optionally specific rules within those grammars, to be deactivated. If no rule is specified, all rules in the grammar are deactivated, including the root rule. The Grammar-Deactivate header MAY be used in both the SET-GRAMMAR and LISTEN methods.
The Hotword header is analogous to the [MRCPv2] Recognition-Mode header, however it has a different name and boolean type in html-speech/1.0 in order to avoid confusion with the Listen-Mode header. When true, the recognizer functions in "hotword" mode, which essentially means that out-of-grammar speech is ignored.
Listen-Mode is used in the LISTEN request to specify whether the recognizer should listen continuously, or return to the idle state after the first RECOGNITION-COMPLETE event. It MUST NOT be used in any other type of request other than LISTEN. When the recognizer is in the listening state, it should include Listen-Mode in all event and status messages it sends.
This header is required to support the continuous speech scenario on the recognizer resource. When sent by the client in a LISTEN or SET-PARAMS request, this header controls whether or not the client is interested in partial results from the service. In this context, the term 'partial' is meant to describe mid-utterance results that provide a best guess at the user's speech thus far (e.g. "deer", "dear father", "dear father christmas"). These results should contain all recognized speech from the point of the last non-partial (ie complete) result, but it may be common for them to omit fully-qualified result attributes like an NBest list, timings, etc. The only guarantee is that the content must be EMMA. Note that this header is valid on both regular command-and-control recognition requests as well as dictation sessions. This is because at the API level, there is no syntactic difference between the recognition types. They are both simply recognition requests over an SRGS grammar or set of URL(s). Additionally partial results can be useful in command-and-control scenarios, for example: open-microphone applications, dictation enrollment applications, and lip-sync. When sent by the server, this header indicates whether the message contents represent a full or partial result. It's valid for a server to send this header on both INTERMEDIATE-RESULT, RECOGNITION-COMPLETE, and in response to a GET-PARAMS messages.
A suggestion from the client to the service on the frequency at which partial results should be sent. It is an integer value represents desired interval expressed in milliseconds. The recognizer does not need to precisely honor the requested interval, but SHOULD provide something close, if it is within the operating parameters of the implementation.
Indicates whether the recognizer is listening or idle. This MUST NOT be included by the client in any requests, and MUST be included by the recognizer in all status and event messages it sends.
Recognition results are often more accurate if the recognizer can train itself to the user's speech over time. This is especially the case with dictation as vocabularies are so large. A User-ID field would allow the recognizer to establish the user's identify if the webapp decided to supply this information.
ISSUE: There were some additional headers proposed few weeks ago: Return-Punctuation; Gender-Number-Pronoun; Return-Formatting; Filter-Profanity. There was pushback that these shouldn't be in a standard because they're very vendor-specific. Thus I haven't included them in this draft. Do we agree these are appropriate to omit?
Speech services MAY support pre-defined grammars that can be referenced through a 'builtin:' uri. For example <builtin:dictation?context=email&lang=en_US>, <builtin:date>, or <builtin:search?context=web>. These can be used as top-level grammars in the Grammar-Activate/Deactivate headers, or in rule references within other grammars. If a speech service does not support the referenced builtin or if it does not specify the builtin in combination with other active grammars, it should return a grammar compilation error.
TODO: perhaps the specific set of grammars should be a MUST for the default built-in user agent, for a certain small set of grammars, but MAY for 3rd-party services. Can't solve this now - but note as an issue to be solved later.
TODO: Note somewhere that vendors are free to support other language model file formats beyond SRGS.
Write some examples of one-shot and continuous recognition, EMMA documents, partial results, vendor-extensions, grammar/rule activation/deactivation, etc.
TODO: recognition from both an audio stream and gesture stream; or recognition from two streams with alternate encodings (one of which is decodable to be listened to by humans, the other is specifically encoded for the recognizer; or recognition from multiple beams of an array mic, with multiple simultaneous users.
C->S: html-speech/1.0 START-MEDIA-STREAM 41021
Resource-ID: recognizer
Audio-codec: audio/dsr-es202212; rate:8000; maxptime:40
Source-Time: 12753248231 (source's local time at the start of the first packet)
S->C: binary audio packet #1 (request-id = 41201 = 1010000011110001)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 0 0 0 0 0 0 0|1 0 0 0 1 1 1 1 0 0 0 0 0 1 0 1|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+
| encoded audio data |
| ... |
| ... |
| ... |
+---------------------------------------------------------------+
S->C: html-speech/1.0 41021 200 IN-PROGRESS (i.e. the service is accepting the audio)
C->S: binary audio packets...
C->S: html-speech/1.0 LISTEN 8322
Resource-Identifier: recognizer
Confidence-Threshold:0.9
Grammar-Activate: <built-in:dictation?context=sms-message>
Listen-Mode: reco-once
Source-time: 12753432234 (where in the input stream recognition should start)
S->C: html-speech/2.0 START-OF-INPUT 8322 IN-PROGRESS
C->S: binary audio packets...
S->C: html-speech/2.0 RECOGNITION-COMPLETE 8322 COMPLETE
C->S: binary audio packets...
C->S: binary audio packet in which the user stops talking
C->S: binary audio packets...
S->C: html-speech/2.0 END-OF-INPUT 8322 IN-PROGRESS (i.e. the recognizer has detected the user stopped talking)
C->S: binary audio packet: end of stream (i.e. since the recognizer has signaled end of input, the UA decides to terminate the stream)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 1 0 0 0 0 0 0|1 0 0 0 1 1 1 1 0 0 0 0 0 1 0 1|0 0 0 0 0 0 0 0|
+---------------------------------------------------------------+
S->C: html-speech/1.0 41021 200 RECOGNITION-COMPLETE (i.e. the service has received the end of stream)
In HTML speech applications, the synthesizer does not participate in the user interface, it merely provides rendered audio upon request, similar to any media server, along with interim events such as marks. The UA buffers the rendered audio, and the application may choose to play it to the user at some point completely unrelated to the synthesizer. It is the synthesizer's role to render the audio stream in a timely manner, at least rapidly enough to support real-time feedback. The synthesizer MAY also render and transmit the stream faster than requied for real time playback, or render multiple streams in parallel, in order to reduce latency in the application. This is a stark contrast to IVR, where the synthesizer essentially renders directly to the user's telephone, and is an active part of the user interface.
TODO: Mention SSML. Use content-type to differentiate between SSML and plain text.
synth-method = "SPEAK"
| "STOP"
| "DEFINE-LEXICON"
The set of synthesizer request methods is a subset of those defined in [MRCPv2]
The SPEAK method operates similarly to its [MRCPv2] namesake. The primary difference is that SPEAK results in a new audio stream being sent from the server to the client, using the same Request-ID. A SPEAK request MUST include the Audio-Codec header. When the rendering has completed, and the end-of-stream message has been sent, the synthesizer sends a SPEAK-COMPLETE event.
When the synthesizer receives a STOP request, it ceases rendering the requests specified in the Active-Request-ID header. If the Active-Request-ID header is missing, it ceases rendring all active SPEAK requests. For any SPEAK request that is ceased, the synthesiser sends an end-of-stream message, and a SPEAK-COMPLETE event.
This is identical to its namesake in [MRCPv2].
Synthesis events are associated with 'IN-PROGRESS' request-state notifications from the synthesizer resource.
synth-event = "INTERIM-EVENT" ; See Interim Events above
| "SPEECH-MARKER" ; An SSML mark has been rendered
| "SPEAK-COMPLETE"
See Interim Events above.
Similar to its namesake in [MRCPv2], except that the Speech-Marker header contains a relative timestamp indicating the elapsed time from the start of the stream.
TODO: this should be sent adjacent to the audio packet at the same time point, so clients can play back in real time.
The same as its [MRCPv2] namesake.
The synthesis headers used in html-speech/1.0 are mostly a subset of those in [MRCPv2], with some minor modification and additions.
synth-header = active-request-id-list
| Completion-Cause
| Completion-Reason
| Voice-Gender
| Voice-Age
| Voice-Variant
| Voice-Name
| Prosody-parameter ; Actually a collection of prosody headers
| Speech-Marker
| Speech-Language
| Failed-URI
| Failed-URI-Cause
| Load-Lexicon
| Lexicon-Search-Order
| Audio-Codec
Audio-Codec = "Audio-Codec:" mime-media-type ; See [RFC3555]
Similar to its namesake in [MRCPv2], except that the clock is defined as starting at zero at the beginning of the output stream. By using a relative time, the UA can calculate when to raise events based on where it is in the playback of the rendered stream.
TODO: insert more synthesis examples
TODO: synthesizing multiple prompts in parallel for playback in the UA when the app needs them
C->S: html-speech/1.0 SPEAK 3257
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-Age:25
Audio-codec:audio/flac
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0">
...
S->C: html-speech/1.0 3257 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=0
S->C: binary audio packet #1 (request-id = 3257 = 110010111001)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 0 0 0 0 0 0 0|1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+
| encoded audio data |
| ... |
| ... |
| ... |
+---------------------------------------------------------------+
S->C: binary audio packets...
S->C: html-speech/1.0 SPEECH-MARKER 3257 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=2059000;marker-1
S->C: binary audio packets...
S->C: html-speech/1.0 SPEAK-COMPLETE 3257 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Completion-Cause:000 normal
Speech-Marker:timestamp=5011000
S->C: binary audio packets...
S->C: binary audio packet: end of stream ( message type = 0x03 )
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| message type | request-id | reserved |
|1 1 0 0 0 0 0 0|1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0|0 0 0 0 0 0 0 0|
+---------------+-------------------------------+---------------+