Executive Summary

Conversational Technologies strongly supports the Extensible MultiModal Annotation 1.0 (EMMA) standard. By providing a standardized yet extensible and flexible basis for representing user input, we believe EMMA has tremendous potential for making possible a wide variety of innovative multimodal applications. In particular, EMMA provides strong support for interoperable applications based on user inputs in human languages in many modalities, including speech, text and handwriting as well as visual modalities such as sign languages. EMMA also supports composite multimodal interactions in which several user inputs in two or more modalities are integrated to represent a single user intent.

The Conversational Technologies EMMA implementations are used in tutorials on commercial applications of natural language processing and spoken dialog systems. We report on two implementations. The first is an EMMA producer (NLWorkbench) which is used to illustrate statistical and grammar-based semantic analysis of speech and text inputs. The second implementation is an EMMA consumer, specifically a viewer for EMMA documents. The viewer can be used in the classroom to simplify examination of EMMA results as well as potentially in commercial applications for debugging spoken dialog systems. In addition, the viewer could also become the basis of an editor which would support such applications as human annotation of EMMA documents to be used as input to machine learning applications. For most of the EMMA structural elements the viewer simply provides a tree structure mirroring the XML markup. The most useful aspects of the viewer are probably the graphical representation for EMMA lattices, the ability to see timestamps as standard dates and the computed durations from EMMA timestamps. The two implementations have been made available as open source software (http://www.sourceforge.net/projects/NLWorkbench).

This is an update to an earlier report (http://lists.w3.org/Archives/Public/www-multimodal/2008Apr/0004.html). The update adds an implementation of the "emma:grammar" annotation and the display of the epsilon transition for lattices. A note has also been added to point out that assertion 801 is inconsistent with the specification.

Deborah Dahl, Conversational Technologies

Technical Details

Schema Validation:

Representative EMMA documents produced by the NLWorkbench for nbest and single semantic results for text and speech inputs successfully validate against the EMMA XML Schema (http://www.w3.org/TR/emma/emma.xsd) using the W3C Schema validator (http://www.w3.org/2001/03/webdata/xsv).

Does not display a no-input result in an easy to understand way Displays nbest in the same order as input document, doesn't reorder. not enforced not enforced The lattice viewer correctly displays epsilon transitions not enforced Lattice viewer doesn't display emma:node. Lattice viewer doesn't display emma:node. This test assertion refers to a configuration that is not allowed by the spec. It should be removed. not enforced not enforced A tree of the information contained in "info" is displayed. Does not display a no-input result in an easy to understand way. This doesn't seem applicable to a viewer. This doesn't seem applicable to a viewer.