- From: White, Jason J <jjwhite@ets.org>
- Date: Thu, 26 May 2022 14:04:49 +0000
- To: Matthew Atkinson <matkinson@tpgi.com>, "public-rqtf@w3.org" <public-rqtf@w3.org>
Matthew's suggested approaches are well considered and move the discussion in the right direction. To support the Accessibility Object Model, it seems to me that a serialization format would have to be defined, and that events for updating the tree would also be needed. I expect this would all have to be specified as a new kind of media resource. Carrying the information as text, with a screen reader running at each end, would also be possible, but there are difficulties. For instance, the output would not be as expected if there were different operating systems or screen readers on each side of the connection. Also, the spoken and braille output would have to be distinguished, and I doubt that events (e.g., navigation commands issued by the user) could easily be transferred between different operating systems and screen reader implementations. For this reasons, I'm initially leaning toward the AOM idea. Further extensions would also be required to address aspects of desktop applications that don't occur in Web contexts. (The APIs allow entire desktops to be shared in the video that is transferred to the client). Perhaps a staged implementation could begin by making browser windows accessible. -----Original Message----- From: Matthew Atkinson <matkinson@tpgi.com> Sent: Thursday, 26 May 2022 6:16 To: public-rqtf@w3.org Subject: [EXTERNAL] Thoughts on accessible web-based screen sharing CAUTION: This email originated from outside of our organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi all, I've been on the periphery of a few conversations recently that have touched on the subject of real-time collaboration, screen sharing and meetings. I had some thoughts about how these use cases could be made more accessible, and we thought it might be worth me posing them to you (apologies if any of these have already been covered). First a brief recap that traditional screen sharing is inaccessible due to sending over a pile of pixels. Back channels have been added to systems like Citrix so that you can have a screen reader running at each end. The remote system can then be explored, because the remote reader has access to its accessibility API. Responsiveness and efficiency is also good, because instead of sending an audio stream from the remote screen reader, essentially only text is being sent over the wire, for the local AT to speak. We've looked at some situations and specs (such as Viewport Capture [1] and its parent Screen Capture [2]) recently, and wondered if these could be made accessible in a similar way. At first I thought, if we are in a web context, how about exposing the DOM (or the part of it that's being shared) from the remote viewport as if it were local? That would work much like the remote access approach above, but you wouldn't need a remote screen reader, because the remote DOM is coming to you. Then I wondered if we could improve on this by building on the Accessibility Object Model (AOM) [3]. As you may know, the AOM is an API for working with the accessibility tree; it allows you to add to and change the accessibility tree without requiring actual corresponding DOM nodes. If using the AOM, we wouldn't even need to transmit a serialisation of the whole DOM; only the relevant subset of it that is the accessibility tree. This would make the whole thing both more efficient and more seamless/robust at the local side. The two main challenges I can see are that more support for AOM than exists now would probably be needed (maybe this would be a good driver for it), and there could be "containment" issues (like potential duplicate IDs on the local side), though perhaps a simple <iframe> would help there. Of course privacy would be a serious consideration; it's possible that information could be exposed in the accessibility tree beyond that broadcast in the shared screen/viewport video stream. Though if we're talking about using this to access one's own remote system, rather than screen sharing, that wouldn't be a concern. What do you think? I'm going to be offline from now until late next week, but wanted to send this off for your consideration. I haven't filed anything in GitHub yet, because it seems way too early, but I'm keen to contribute to any conversations arising from this, if you think it's worth exploring. best regards, Matthew [1] https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-viewport%2F&data=05%7C01%7Cjjwhite%40ets.org%7C5bd00ccee0fc41a51cf508da3f00d2ca%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637891570028331183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UTTbGz6EaeQIq%2FQmPP2YHX6F8FuV8Czb24z50lgHwlM%3D&reserved=0 [2] https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.org%2FTR%2Fscreen-capture%2F&data=05%7C01%7Cjjwhite%40ets.org%7C5bd00ccee0fc41a51cf508da3f00d2ca%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637891570028331183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qWexNkbg5Gf%2Fo5Ou8jfYOwpfUMgt14r4y%2BzyTpXaK38%3D&reserved=0 [3] https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwicg.github.io%2Faom%2Fexplainer.html&data=05%7C01%7Cjjwhite%40ets.org%7C5bd00ccee0fc41a51cf508da3f00d2ca%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637891570028331183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w%2Fnlh4Khi%2F572I3eIDzhuAXPw3%2B%2FauoFKqrhgrJJ25A%3D&reserved=0 -- Matthew Tylee Atkinson (he/him) -- Senior Accessibility Engineer TPG Interactive https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.tpgi.com%2F&data=05%7C01%7Cjjwhite%40ets.org%7C5bd00ccee0fc41a51cf508da3f00d2ca%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637891570028331183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HhIIYndsG92PuZavppnN6drrQpbtP%2ByNMBtSmb3dDlw%3D&reserved=0 A Vispero Company https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.vispero.com%2F&data=05%7C01%7Cjjwhite%40ets.org%7C5bd00ccee0fc41a51cf508da3f00d2ca%7C0ba6e9b760b34fae92f37e6ddd9e9b65%7C0%7C0%7C637891570028331183%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eLeyQxOiJ3TVQKRdbmxg7Z5czvuljZNO64iWcnwEuHk%3D&reserved=0 -- This message is intended to be confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please delete this message from your system and notify us immediately. Any disclosure, copying, distribution or action taken or omitted to be taken by an unintended recipient in reliance on this message is prohibited and may be unlawful. ________________________________ This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance. ________________________________
Received on Thursday, 26 May 2022 14:05:12 UTC