RE: Dialogue-based Activities in Digital Textbooks

Dirk,

Thank you.

In addition to self-studying scenarios, where dialogue-based activities might resemble “for more information”, there are also dialogue-based assessment scenarios to consider. Dialogue-based activities could be assigned by teachers, serving as exercises, activities, or even quizzes.

Regarding dialogue-system-to-digital-textbook interoperation, as learners answer questions incorrectly, e.g., during exercises or activities, dialogue systems could turn to pages in digital textbooks and highlight hypertext content or seek to video content for learners to review.

Q&A systems are also possible where dialogue systems could answer learners’ questions while referring to textbook or external content. Hyperlinks to such Q&A dialogue-based activities – where learners ask questions – could be placed inline, e.g., in textbook margins, as well as at the ends of sections and chapters.

The points where learners want to ask for clarifications regarding specific content in digital textbooks, at scale, sounds like interesting educational data.

Digital textbooks and dialogue systems could also interoperate with teachers and their software in new ways. Perhaps learners could leave “voicemail” or “voice-based email” for teachers or teachers’ assistants. Teachers and their assistants could be fallbacks for dialogue systems in some cases. This could be useful when a Q&A system does not understand a question, does not have an answer, or is otherwise programmed or designed to route to a teacher.


Best regards,
Adam


From: dirk.schnelle@jvoicexml.org <dirk.schnelle@jvoicexml.org>
Sent: Friday, July 2, 2021 3:45:00 PM
To: Adam Sobieski <adamsobieski@hotmail.com>
Cc: public-voiceinteraction@w3.org <public-voiceinteraction@w3.org>
Subject: Re: Dialogue-based Activities in Digital Textbooks

Dear Adam,

Welcome to the group.


This sounds like some interesting thoughts about engaging users in what they are currently reading or studying.

I am just thinking if there are really clear borders between the digital text book and the conversational agent.

Is it as you described that the primary part would be the digital textbook and the conversational agent only comes into play if the user explicitly asks for it? I would name that a self studying use case.

Could it be also driven by the conversational agent with some digital textbook called by the conversational agent and with the option of the "reader" to ask for clarification? I would name that a teacher use case.

Dirk

Am 25.06.2021 07:34 schrieb Adam Sobieski <adamsobieski@hotmail.com>:

W3C Voice Interaction Community Group,



Hello. I am new to the group and would like to share some thoughts about dialogue-based activities in digital textbooks.



Introduction



Consider a user reading a digital textbook. The digital textbook includes dialogue-based educational activities. These activities, e.g., educational dialogues about specific concepts discussed in the digital textbook, could be provided throughout the digital textbook.



The digital textbook’s publisher desires for these dialogue-based activities to work with any conversational assistants that users might have. The digital textbook’s publisher might also want to provide services which can interoperate with users’ conversational assistants.



Launching Dialogue-based Activities from Conversational Assistants



Here, the user is reading the digital textbook, comes across a dialogue-based activity, and thinks to launch the dialogue-based activity via a verbal interaction with a conversational assistant. “Siri, begin activity #123 in my science textbook.” Or, “Siri, begin activity #123 in this book.” Or, “Siri, begin activity #123.”



Understanding these utterances appears to require some advanced capabilities on the part of a conversational assistant.



Conversational assistants could learn about which courses users are enrolled in, which textbooks they are using, and, perhaps, to be able to process the syllabi of the courses. These capabilities would be of use when resolving verbal references like “my science textbook”.



Conversational assistants could jointly attend to the digital textbooks in use. Conversational assistants could process the content of users’ active documents, scanning for dialogue-based activities, e.g., after asking permission to do so. These capabilities could be of use when resolving verbal references such as “this book”, or when understanding utterances where the particular digital textbook is not mentioned. These capabilities could be of use when the user wants their conversational assistant to be aware of their working context on a device.



Launching Dialogue-based Activities from Digital Textbooks



Here, the user is reading the digital textbook and clicks on a hyperlink which results in sending data to a conversational assistant about the particular dialogue-based activity.



To achieve this, one could register a new URL scheme or a new MIME type as being handled by an application which interoperates with conversational assistants.



For instance, a new URL scheme could resemble: dialogue+activity://specific.publisher.org/textbook/edition/activity-123 .



Clicking on certain hyperlinks in digital textbooks would result in inter-process, potentially inter-device, communication between Web browsers and conversational assistants.



As envisioned, after users clicked on such hyperlinks, applications handling the URL scheme or the MIME type would present users with menus of available conversational assistants as needed (e.g., one on a tablet computer and one on a proximate networked smart speaker device). Users would select which conversational assistant that they desired to interact with. Applications handling the URL scheme or MIME type would, then, provide selected conversational assistants with data for launching dialogue-based activities. Upon successful completion of the multi-process, potentially multi-device, operation, conversational assistants could speak to users, indicating readiness and initiating conversation.



Interoperation Scenarios



Digital textbooks could use JavaScript for bidirectional communication with remote services which could interoperate with conversational assistants.



Conversational assistants would, then, be able to turn to pages and highlight content in digital textbooks for users.



Conversational assistants would, then, also be able to indicate successful completions of dialogue-based activities to digital textbooks.





Best regards,

Adam Sobieski

http://www.phoster.com



P.S.: In addition to educational scenarios, the technologies discussed above could enable business scenarios. There could be hyperlinks on a webpage for users to click on to launch dialogue-based activities about products.