Dialogue-based Activities in Digital Textbooks

W3C Voice Interaction Community Group,

Hello. I am new to the group and would like to share some thoughts about dialogue-based activities in digital textbooks.

Introduction

Consider a user reading a digital textbook. The digital textbook includes dialogue-based educational activities. These activities, e.g., educational dialogues about specific concepts discussed in the digital textbook, could be provided throughout the digital textbook.

The digital textbook’s publisher desires for these dialogue-based activities to work with any conversational assistants that users might have. The digital textbook’s publisher might also want to provide services which can interoperate with users’ conversational assistants.

Launching Dialogue-based Activities from Conversational Assistants

Here, the user is reading the digital textbook, comes across a dialogue-based activity, and thinks to launch the dialogue-based activity via a verbal interaction with a conversational assistant. “Siri, begin activity #123 in my science textbook.” Or, “Siri, begin activity #123 in this book.” Or, “Siri, begin activity #123.”

Understanding these utterances appears to require some advanced capabilities on the part of a conversational assistant.

Conversational assistants could learn about which courses users are enrolled in, which textbooks they are using, and, perhaps, to be able to process the syllabi of the courses. These capabilities would be of use when resolving verbal references like “my science textbook”.

Conversational assistants could jointly attend to the digital textbooks in use. Conversational assistants could process the content of users’ active documents, scanning for dialogue-based activities, e.g., after asking permission to do so. These capabilities could be of use when resolving verbal references such as “this book”, or when understanding utterances where the particular digital textbook is not mentioned. These capabilities could be of use when the user wants their conversational assistant to be aware of their working context on a device.

Launching Dialogue-based Activities from Digital Textbooks

Here, the user is reading the digital textbook and clicks on a hyperlink which results in sending data to a conversational assistant about the particular dialogue-based activity.

To achieve this, one could register a new URL scheme or a new MIME type as being handled by an application which interoperates with conversational assistants.

For instance, a new URL scheme could resemble: dialogue+activity://specific.publisher.org/textbook/edition/activity-123 .

Clicking on certain hyperlinks in digital textbooks would result in inter-process, potentially inter-device, communication between Web browsers and conversational assistants.

As envisioned, after users clicked on such hyperlinks, applications handling the URL scheme or the MIME type would present users with menus of available conversational assistants as needed (e.g., one on a tablet computer and one on a proximate networked smart speaker device). Users would select which conversational assistant that they desired to interact with. Applications handling the URL scheme or MIME type would, then, provide selected conversational assistants with data for launching dialogue-based activities. Upon successful completion of the multi-process, potentially multi-device, operation, conversational assistants could speak to users, indicating readiness and initiating conversation.

Interoperation Scenarios

Digital textbooks could use JavaScript for bidirectional communication with remote services which could interoperate with conversational assistants.

Conversational assistants would, then, be able to turn to pages and highlight content in digital textbooks for users.

Conversational assistants would, then, also be able to indicate successful completions of dialogue-based activities to digital textbooks.


Best regards,
Adam Sobieski
http://www.phoster.com

P.S.: In addition to educational scenarios, the technologies discussed above could enable business scenarios. There could be hyperlinks on a webpage for users to click on to launch dialogue-based activities about products.