RE: [EXTERNAL] Natural language interfaces and conversational agents

Yes, we could focus on the search interaction if it involves natural language processing, but it’s also possible to achieve the same with a more complex “smart agent”. For example, I read an article somewhere which asserted that Microsoft’s smart agent was expected to be integrated more closely for use as an assistant with their Microsoft Office products. That’s obviously just an example of what it is possible to implement.
I suspect we’re principally interested in the more complex natural language processing applications than processing of search queries. For example, there’s a “voice search” option in one of my Web browsers that I assume comes from google.com, and what we’re writing could be applicable to that kind of interface – even if it isn’t a paradigm case of what we’re working on.
It still seems to me that the natural language interaction aspect truly is at the core of what we’re striving for as the scope of the work, but that we’ll also need to say something informative and coherent about the larger context in which it operates.

From: John Paton <John.Paton@rnib.org.uk>
Sent: Friday, 5 March 2021 11:09
To: White, Jason J <jjwhite@ets.org>; Janina Sajka <janina@rednote.net>
Cc: Joshue O'Connor <joconnor@w3.org>; Scott Hollier <scott@hollier.info>; public-rqtf@w3.org
Subject: RE: [EXTERNAL] Natural language interfaces and conversational agents

Hi Jason,

With the example of a word processor that you give however, after the initial action of typing in a query the user normally chooses from a selection of potential options which to my understanding brings that interface out of scope of this work. Would this work focus entirely on the user requirements of that initial search box?

Cheers,

John

From: White, Jason J <jjwhite@ets.org>
Sent: 05 March 2021 16:00
To: Janina Sajka <janina@rednote.net>
Cc: Joshue O'Connor <joconnor@w3.org>; John Paton <John.Paton@rnib.org.uk>; Scott Hollier <scott@hollier.info>; public-rqtf@w3.org
Subject: RE: [EXTERNAL] Natural language interfaces and conversational agents

CAUTION: External. Do not click links or open attachments unless you know the content is safe.
________________________________
Thank you, Janina, for raising important considerations here. Besides the "hands-off" nature, another advantage is efficiency. Consider a complex application such as a modern word processor. In my experience, it's often faster to type a natural language query (into the application itself or into a Web search engine) to find what one is looking for than to traverse a complex structure of graphical menus and dialogues.
Microsoft Office applications let you type a keystroke (Alt-Q, if I remember rightly) and enter a search query to find what you are looking for. At least for a screen reader user with a keyboard interface, that's often faster than having to search menus/dialogues.
Thus I suspect that even if a natural language interface is hands-on (requires touch/keyboard input for the particular user) there are interfaces for which it's still going to be more efficient than to switch to a GUI-based alternative. Efficiency matters in the workplace, in education, and elsewhere, even if the alternative facilitation is technically "accessible" and could still be used.
Of course, there's also the case in which the same functionality is not provided via any other user interface than natural language. However, that isn't the case Janina describes.

-----Original Message-----
From: Janina Sajka <janina@rednote.net<mailto:janina@rednote.net>>
Sent: Friday, 5 March 2021 10:15
To: White, Jason J <jjwhite@ets.org<mailto:jjwhite@ets.org>>
Cc: Joshue O'Connor <joconnor@w3.org<mailto:joconnor@w3.org>>; John Paton <John.Paton@rnib.org.uk<mailto:John.Paton@rnib.org.uk>>; Scott Hollier <scott@hollier.info<mailto:scott@hollier.info>>; public-rqtf@w3.org<mailto:public-rqtf@w3.org>
Subject: Re: [EXTERNAL] Natural language interfaces and conversational agents

I've been sitting back and reading this conversation, and I'm appreciating some good progress in scoping potential work.

I want to play the devil's advocate a bit.

I'm unconvinced it's helpful to say the universe of natural language interfaces needs to be accessible to all pwd, because some of the edge cases are, imo, too cumbersome to pursue when alternative technologies exist that those users might find far more attractive. This is the concept known in U.S. disability regulation as "equivalent facilitation."

If one can achieve the same functionality in a more usable interface, why would we insist they should work harder to do so in natural language environments?

Seems to me the key function of natural language interfaces is not the language, but the hands off nature of the interaction. One is able to work with the technology without recourse to keyboard or mouse, and able to do so while one's hands may be otherwise occupied. Thus, one is able to wkr even from some physical distance. To my mind this explains the recent success of these agents, and it explains why the same general idea went nowhere in earlier days--think of the old MIT natural language phonomen Eliza:

https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FELIZA&amp;data=04%7C01%7Cjjwhite%40ets.org%7C5f3db94e4ac443afd88308d8dfe96fe5%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505540989980996%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=14VYqmQ0pQug%2F%2FY3ju%2FPlOF9ZHuDiy5YNLBj1OcDbac%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FELIZA&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708904801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=C%2F6ravuk6UPBz8V4%2BDxiBPDsk6mzUtw5unmUIk2Vcmk%3D&reserved=0>

I suggest we need to find a way to honor suitability for purpose into our scoping somehow.

Best,

Janina

White, Jason J writes:
> Thank you, Josh, for your thoughtful commentary. I think everyone agrees there are challenging scope boundaries in this area that we haven’t yet resolved. A good example of the problem that I’ve read in the research literature is as follows.
> Consider a navigation application with a natural language interface and a graphically displayed map. The user points to a place on the map and says “send the ambulance here” to the voice agent. It’s the combination of the utterance and the pointing that determines what the user is referring to, but the pointing gesture and the map aren’t strictly speaking part of the natural language aspect of the design.
>
> From: Joshue O'Connor <joconnor@w3.org<mailto:joconnor@w3.org>>
> Sent: Friday, 5 March 2021 9:30
> To: White, Jason J <jjwhite@ets.org<mailto:jjwhite@ets.org>>
> Cc: John Paton <John.Paton@rnib.org.uk<mailto:John.Paton@rnib.org.uk>>; Scott Hollier
> <scott@hollier.info<mailto:scott@hollier.info>>; public-rqtf@w3.org<mailto:public-rqtf@w3.org>
> Subject: Re: [EXTERNAL] Natural language interfaces and conversational
> agents
>
> Hi Jason and all,
>
> White, Jason J wrote on 05/03/2021 13:45:
>
> • Focus our work on the accessibility of the natural language interaction itself. As far as I know, no one has documented the accessibility requirements for it elsewhere.
> • Refer to other guidance (WCAG, XAUR, RAUR, etc.) for the accessibility of other aspects of the user interface.
> • Note that natural language interaction can occur as part of a larger interface and that the whole interface needs to be accessible.
>
> +1 from me, with qualifying comments to signal to you all my (ever) shifting perspective on this. As I commented in a private mail to Jason, the situation we are in regarding scope, and various challenges can be broadly broken into:
>
> 1) The I/O aspect
> 2) The service (or agent) behind it
>
> There are also options on these approaches/perspectives, on these aspects which are 'narrow' - focusing initially on Speech/Voice User Interfaces only or much broader. My two cents are that starting from the narrow perspective would give us a basis to add other modalities later on, but there is push back on that, which I also appreciate and understand. If we were to then take the broader approach and try to widen the scope we can get into very muddy and indistinct water super quickly. For the broader scope approach my current thinking is that we may avoid confusion, mixing streams etc if we took up the idea of 'Natural Language Interface Accessibility User Requirements'. Thinking of Michael's sensible suggestion to have clearly defined terms etc this one if my fave, as it is already well defined, isn't just a marketing term etc. I prefer this to Smart Agents, which potentially pushes us into a sea of IoT and related services. One one level this may not be a bad thing, but we don't have infinite time either.
>
> To me, if we want to realise a user requirements document with a broader scope - this really nicely covers the need for a multi-modal, device independent descriptor for the I/O side and we can add a strapline or <h2> etc saying ' Accessibility infrastructure and supporting services' or similar. I'm thinking this would allow us to cover VUIs, and other I/O modalities for other groups, the kind of things that Jason refers to as 'Conversational' etc as well as look at the services behind them.
>
> This is really helpful Jason, and please lets continue to discuss these options, and indeed any more we may be missing. If we were to go down the broader road, then I find this terminology is the most suitable nomenclature that I've seen yet.
>
> HTH
>
> Josh
> --
> Emerging Web Technology Specialist/Accessibility (WAI/W3C)
>
> ________________________________
>
> This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
>
>
> Thank you for your compliance.
>
> ________________________________

--

Janina Sajka
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fin%2Fjsajka&amp;data=04%7C01%7Cjjwhite%40ets.org%7C5f3db94e4ac443afd88308d8dfe96fe5%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505540989990947%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gE2cKdz1Zc2yo0qq0ZbZITGV7TqBh7QqFVOqAAzoGoE%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fin%2Fjsajka&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708914759%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IMofiFyY8gMHh7SXBIWL0VDhPIdK6pS6bk1L%2FvfFw9E%3D&reserved=0>

Linux Foundation Fellow
Executive Chair, Accessibility Workgroup:https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fa11y.org%2F&amp;data=04%7C01%7Cjjwhite%40ets.org%7C5f3db94e4ac443afd88308d8dfe96fe5%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505540989990947%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=kZbR2X3J%2BEUYTvZa41lCf5lcRIto4ZwmzqNr%2FcvTJLs%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fa11y.org%2F&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708914759%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FrLVVAqZbZEHZVHAbNVFZ%2FNWaZblTYVhiP9I%2FseMSX0%3D&reserved=0>

The World Wide Web Consortium (W3C), Web Accessibility Initiative (WAI)
Co-Chair, Accessible Platform Architectureshttps://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.w3.org%2Fwai%2Fapa&amp;data=04%7C01%7Cjjwhite%40ets.org%7C5f3db94e4ac443afd88308d8dfe96fe5%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505540989990947%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=KIVPMylYRMJSdBnhdcoaipWhRufG4OaVarUzt%2FzoKA0%3D&amp;reserved=0



________________________________

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.


Thank you for your compliance.

________________________________

--
[Image removed by sender. RNIB Take on 250 Logo]<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rnib.org.uk%2Fdonations-and-fundraising%2Fchallenge-events%2Ftake-250-rnib&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708924720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QHS%2BnufSFHV24%2FZwOjwL6%2F7jt3DKxughwmLb3OJmIik%3D&reserved=0>

Every day, 250 people in the UK begin to lose their sight, that’s why we need you to Take on 250 for RNIB<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rnib.org.uk%2Ftake-on-250&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708924720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dkdlE97ujgSzp9lFQ43gC2iH2hMLKoM%2B3VcrWNshrHQ%3D&reserved=0>. Walking, running, cycling or swimming; baking, singing, dancing or knitting. It’s all up for grabs – and you complete 250 of whatever you decide.
Join Us<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rnib.org.uk%2Ftake-on-250&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708934670%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2Bxys1W7oxBq6sPbsnUVF1VKFxc4DMXyintgh%2FN0dhE%3D&reserved=0> and make a difference for people facing sight loss.

--

DISCLAIMER:

The information contained in this email and any attachments is confidential and may be privileged. If you are not the intended recipient you should not use, disclose, distribute or copy any of the content of it or of any attachment; you are requested to notify the sender immediately of your receipt of the email and then to delete it and any attachments from your system.

RNIB endeavours to ensure that all emails and attachments are virus free. We cannot, however, guarantee nor accept any responsibility for the integrity of unsecure email.

We therefore recommend that you use up to date anti-virus software and scan all communications.

Please note that the statements and views expressed in this email and any attachments are those of the author and do not necessarily represent those of RNIB.

RNIB Registered Charity Number: 226227

Website: https://www.rnib.org.uk<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rnib.org.uk%2F&data=04%7C01%7Cjjwhite%40ets.org%7Cf046c2a0a62b4f4df58308d8dff10d5b%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637505573708934670%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=loFg38NtmPmrmhv%2BTcPFUTcDcvUxRZGkyB%2FOQdaUjVA%3D&reserved=0>

________________________________

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.


Thank you for your compliance.

________________________________