RE: [Cloud Browser] Architecture updates/questions

Dear John,

Thank you for providing the Shadow DOM document.
I will go through the document to understand and estimate the problems we are going to face. I will surely get back to you with some questions.

Mit freundlichen Grüßen / Viele Grüße / Best Regards
Alexandra Mikityuk


DEUTSCHE TELEKOM AG
T-Labs (Research & Innovation)
Alexandra Mikityuk
Winterfeldtstr. 21, 10781 Berlin
+4930835358151 (Tel.)
+4930835358408 (Fax)
E-Mail: alexandra.mikityuk@telekom.de
www.telekom.com<http://www.telekom.com/>

Erleben, was verbindet.

Die gesetzlichen Pflichtangaben finden Sie unter: www.telekom.com/pflichtangaben<http://www.telekom.com/pflichtangaben>

Grosse Veränderungen fangen klein an – Ressourcen schonen und nicht jede E-Mail drucken.


From: John Foliot [mailto:john.foliot@deque.com]
Sent: Freitag, 6. Mai 2016 18:45
To: Meerveld, Colin
Cc: Mikityuk, Alexandra; public-web-and-tv@w3.org
Subject: Re: [Cloud Browser] Architecture updates/questions

> Such a function set could look like:
> Composition; execution; UI rendering; Video rendering; encoding/ decoding (hardware,
> software); scrambling/ descrambling (hardware, software); networking; session mgmt.;
> user input processing; transcoding.

Hi Alexandra,

I'm struggling here to understand where DOM representation and access fits into the mix.

For example, you note UI rendering (but is that just the visual UI, or does that encompass exposing the DOM structure as well?) and user input processing (but does that take into account alternate input modes? alternate input devices?)

Browsers that are tied to hardware solutions also act as a conduit to the Accessibility APIs of the various operating systems (see: https://www.w3.org/TR/html-aam-1.0/) so that the abstraction of both input and output can be communicated to Assitive Technologies as required. (This is why ARIA is so important, as it allows for custom widgets to be accessible to AT, by allowing a means to express role, state, and property of any widget on the page in an abstract way - for example a volume control slider set to 8 (role="slider", property="volume control", state="8") - see here: https://www.w3.org/WAI/GL/wiki/Using_WAI-ARIA_state_and_property_attributes_to_expose_the_state_of_a_user_interface_component)

I will suggest that one thing you will require is a method to connect alternative input/output devices to some part of the client's hardware stack, so that (for example) audio can be converted to braille output, or a person with mobility issues can interact with the content using non-traditional devices (http://www.cetfund.org/files/IMG_0439_JPG.jpg). Often times that will also involve the need for additional non-browser software (screen reader for example), and it is unclear where that would live in the ecosystem.

*****************

Also, I read here:

The Graphics Library renders and writes the data into the Buffer.

However, in the case of the Cloud Browser, the data is not sent to the display.

The data is encoded using a required video codec and sent as a video stream to the Cloud Browser client over IP.

From an accessibility perspective, I think this is going to be your biggest issue with a "cloud" browser, as it is significantly more than just sending data as a video codec stream to a screen.

I have mentioned previously that this sounds very similar to the early-days problems with the <canvas> element in HTML5, where again we had a method of generating moving pixels to a screen, but no "bones" or structure behind that for Assistive Technology to interact with. The "solution" was the creation of a Shadow DOM (https://www.w3.org/TR/shadow-dom/) and I strongly recommend this group review that document for a clearer explanation of the problem, and the proposed W3C recommendation to address the problem. I'm not sure if this is 100% applicable (or achievable in the envisioned ecosystem), but I will also suggest that this is critical for the cloud browser initiative moving forward.

JF

On Fri, May 6, 2016 at 9:06 AM, Meerveld, Colin <C.Meerveld@activevideo.com<mailto:C.Meerveld@activevideo.com>> wrote:
Hi Alexandra,

Thank you for this update, although i am a bit confused. It seems that you suggest that there are 4 independent architectures. I believe they are tight together. For example the Cloud player approach is a result of the Single stream Approach. They are different perspectives on the same architecture (The first from the player and the latter from the transport). In addition i am not sure if we have to define a flexible architecture. Instead of flexible i would rather see a solution where a implementer could use a subset and/or constrains. For example a rte which only support a single stream or a rte which only allow drm protected double stream media.

I do like a explanatory description per approach but mainly as example rather than the only solution. In your example you could argue if it needs to be IP or the need of HTTP/RFB.

regards,

Colin Meerveld





On 25 Apr 2016, at 20:32, Alexandra.Mikityuk@telekom.de<mailto:Alexandra.Mikityuk@telekom.de> wrote:

Dear all,

working on the action points [1] to [5] with regard to updates of the Cloud Browser architecture, I have a couple of questions to the group.

1.      On the one side it would be sufficient to define a flexible architecture, where the functions are not hard-coded and are rather assigned on the fly depending on the approach.
However, as we have already defined 4 approaches and agreed that they would be enough for the moment to cover the Use Cases we have [6],
my 1st question would be, would it make sense to define a simple function set and describe which functions are supported by the server and which by the client for each of the
approaches?
Such a function set could look like:
Composition; execution; UI rendering; Video rendering; encoding/ decoding (hardware, software); scrambling/ descrambling (hardware, software); networking; session mgmt.; user input processing; transcoding.

By defining of these functions for each Approach, it would be more clear, which of these build the Runtime Environment of the Cloud Browser Client.
Some of these would be done only in hardware in case of the e.g. Single Stream Cloud Player

2.      The second question would be with regard to the description of these approaches. Would it make sense to describe it with following sections
•        Technical description
•        Considered devices
•        Operation scope (for which cases is used)

To make it more clear, I have made an exemplarily description of the Single Stream Cloud Player approach that you can find in the end of the E-Mail in [12].

3.      Would it be better to include the Media Player into the Runtime Environment – this way it would be more clear that the Media Player is a part of the Runtime Environment of the client.
I have also provided more clear arch pictures, inspired by the Initial Concept arch made by Colin. Please, see [7] to [10]. Do they provide a better description or are they too technical and it would be better to stay with the current once available in [11]?


[1] https://www.w3.org/2011/webtv/track/actions/215

[2] https://www.w3.org/2011/webtv/track/actions/220

[3] https://www.w3.org/2011/webtv/track/actions/221

[4] https://www.w3.org/2011/webtv/track/actions/225

[5] https://www.w3.org/2011/webtv/track/actions/227

[6] https://www.w3.org/2011/webtv/wiki/Main_Page/Cloud_Browser_TF/UseCases

[7] https://www.w3.org/2011/webtv/wiki/File:Ss-cp.png

[8] https://www.w3.org/2011/webtv/wiki/File:Ss-lp.png

[9] https://www.w3.org/2011/webtv/wiki/File:Ds-cp.png

[10] https://www.w3.org/2011/webtv/wiki/File:Ds-lp.png

[11] https://www.w3.org/2011/webtv/wiki/Main_Page/Cloud_Browser_TF/Architecture


[12] Description example:

Technical Description:
After the input data is downloaded, the CSS and HTML is parsed and interpreted by the Cloud Browser.
The JavaScript is processed and executed by the JavaScript engine of the browser.
After the DOM and the Render Trees are built, the painting commands are sent to the Graphics Library over the Graphics Context.
The video is downloaded from the Media Sources by the browser and is also sent to the Graphics Library of the CB server.
The Graphics Library renders and writes the data into the Buffer.
However, in the case of the Cloud Browser, the data is not sent to the display.
The data is encoded using a required video codec and sent as a video stream to the Cloud Browser client over IP.
For this delivery either HTTP or RFB application layer protocols could be used, depending on the implementation.
The CB client receives the input stream that is directly decoded by the Decoder.
The stream might be also first processed by the Descrambler, if the content is encrypted by the CB with an encryption scheme supported by the CB client.

Considered devices: legacy devices, low-end and low-power devices only with hardware functions.

Operation Scope:
In this approach the CB overtakes all CB client software functionalities.
The CB adapts the output data to the available hardware of the CB client, e.g. secure processor, video decoding, network.
Therefore, the client functions are limited to the available hardware.
The reduced Runtime Environment of the CB client is limited to support of these legacy hardware capabilities, if required.
It could contain specific video functions that could not be reduced, as quality probes for quality measurements.
This approach could be used e.g. for simplification of the client decryption.
The decryption functions are terminated in the Cloud and re-encrypted by sending the stream down to the client.
This Scenario implies decryption capabilities of the Cloud Environment.
The client doesn't need to support the source encryption.


Mit freundlichen Grüßen / Viele Grüße / Best Regards
Alexandra Mikityuk


DEUTSCHE TELEKOM AG
T-Labs (Research & Innovation)
Alexandra Mikityuk
Winterfeldtstr. 21, 10781 Berlin
+4930835358151<tel:%2B4930835358151> (Tel.)
+4930835358408<tel:%2B4930835358408> (Fax)
E-Mail: alexandra.mikityuk@telekom.de<mailto:alexandra.mikityuk@telekom.de>
www.telekom.com<http://www.telekom.com/>
ERLEBEN, WAS VERBINDET.

Die gesetzlichen Pflichtangaben finden Sie unter: www.telekom.com/pflichtangaben<http://www.telekom.com/pflichtangaben>

GROSSE VERÄNDERUNGEN FANGEN KLEIN AN – RESSOURCEN SCHONEN UND NICHT JEDE E-MAIL DRUCKEN.




--
John Foliot
Principal Accessibility Consultant
Deque Systems Inc.
john.foliot@deque.com<mailto:john.foliot@deque.com>

Advancing the mission of digital accessibility and inclusion