[w3c/editing] Relationship with `VisualViewport` (#325)

Porting over from MSEdgeExplainers repo https://github.com/MicrosoftEdge/MSEdgeExplainers/issues/389

@alice filed:

> Coming here from the [TAG review thread](https://github.com/w3ctag/design-reviews/issues/507).
>
> In terms of solving the general problem of informing the web page about occlusions, it does seem like `VisualViewport` is trying to solve some of the same problems, but is extremely limited (for example, it assumes the visual viewport is a rectangle).
>
> I'm a little lost as to whether `VisualViewport` is intended to account for whether or not the virtual keyboard is overlaying page content or not - to me it reads as if it assumes the keyboard is overlaying page content, since otherwise it will be the same as the layout viewport.
>
> These comments of Maciej's strike a chord:
>
>  I think Visual Viewport API could be enhanced to report individual overlay rects. That way, authors who want to adapt to available screen space won't have to juggle two different APIs. And we won't need to invent still other new APIs for other forms of onscreen overlays, like PIP, or the iPadOS Universal Callout Bar which appears even with a hardware keyboard attached.
>
> Indeed, the "geometrychange" event is very similar to the "onresize" event on `VisualViewport`.
> 
> It would be a shame to miss an opportunity to design a general purpose API for these various types of screen occlusions. I wonder whether `VisualViewport` could contain a collection of objects with a common interface, of which `virtualKeyboard` could be one.
>
> We would still need to figure out how to expose useful CSS environment variables for each one, or indeed if environment variables are the best way forward. I think the CSSWG may have some thoughts there.

@BoCupp-Microsoft replied:

> > In terms of solving the general problem of informing the web page about occlusions, it does seem like VisualViewport is trying to solve some of the same problems, but is extremely limited (for example, it assumes the visual viewport is a rectangle).
> 
> The VisualViewport only solves the problem of knowing when the visual viewport is changing.  It is not a general purpose reporter of occlusions and provides no information today about occlusions.  I mean specifically that things which overlay the visual viewport don't change the visual viewport and therefore don't report any events via the VisualViewport API.  Examples of things overlaying the visual viewport and therefore not affecting it include a floating keyboard, overlapping windows on the desktop, and picture-in-picture.
> 
> The user can tell when something changes the size of the visual viewport (and the layout viewport is not updated to match) because it will result in user perceivable "slop" that allows the layout viewport to be panned around in a smaller visual viewport.  Additionally, part of the layout viewport will not be visible to the user (at least not without a change in scale).
> 
> In contrast, when a window overlaps and doesn't resize the visual viewport, no such "slop" will exist.  It will be on the author to trigger an update to the layout of the page if they want content to be repositioned (assuming that there is some event to inform them of this occlusion).
> 
> In general I don't think we want to report all the things layered over top of the visual viewport.  I wouldn't want web pages updating their layout as I ALT+TAB to another application that overlaps my browser, or have a web page fight me as a drag around a floating keyboard or a picture-in-picture window.
> 
> The "docked" virtual keyboard is a special case.  When it appears, users do expect to be able to see what they're typing, but authors may not want the experience we offer by default.  L-shaped viewports aside, adjusting the visual viewport means we'll be creating the "slop" I describe above and jumping part of the page out of the visual viewport to ensure the user can see the blinking caret.  If I'm an author that has created a web application whose UI fits in 100vw x 100vh, I might rather adjust the height of one of my subscrollers rather than "unglue" my UI so that the user can start panning it around.
> 
> The VirtualKeyboard API allows the author to handle that special case by declaring that the virtual keyboard overlay the browser window instead of changing the size of its visual viewport.  Since the size of the visual viewport won't be changing in this case, I think its inappropriate for it to be dispatching any events describing that overlay.
> 
> > I'm a little lost as to whether `VisualViewport` is intended to account for whether or not the virtual keyboard is overlaying page content or not - to me it reads as if it assumes the keyboard is overlaying page content, since otherwise it will be the same as the layout viewport.
> 
> I'm unsure exactly what you mean when you say account for it.  Maybe I answered your question already above?  If not, help me understand what you'd like to know and I can probably write a better response.
> 
> >These comments of Maciej's strike a chord:
> >>I think Visual Viewport API could be enhanced to report individual overlay rects. That way, authors who want to adapt to available screen space won't have to juggle two different APIs. And we won't need to invent still other new APIs for other forms of onscreen overlays, like PIP, or the iPadOS Universal Callout Bar which appears even with a hardware keyboard attached.
> >>
> >Indeed, the "geometrychange" event is very similar to the "onresize" event on VisualViewport.
> 
> I disagree on two points: one I explained above about there being no need for a general purpose overlay API, and the other is the notion that its inherently bad to create a more specialized API surface for something as important as the virtual keyboard.  You are likely aware of this [complementary proposal](https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardPolicy/explainer.md) that allows authors to hide and show the virtual keyboard.  Creating a separate VirtualKeyboard interface allows for a cohesive API surface where all things virtual keyboard related can be gathered.
> 
> > We would still need to figure out how to expose useful CSS environment variables for each one, or indeed if environment variables are the best way forward. I think the CSSWG may have some thoughts there.
> 
> I'll write a separate response for this point soon.

@BoCupp-Microsoft followed up with:

> @alice the explainer has recently been updated and includes a proposal for CSS environment variables along with examples showing how to apply them.

@snianu then closed the issue with:

> TAG reviewed the explainers and the discussions we had in TPAC and [supported ](https://github.com/w3ctag/design-reviews/issues/498#issuecomment-729466869)the proposal.

@cythia replied:

> Just to clarify - we were happy with the show/hide policy. (https://github.com/w3ctag/design-reviews/issues/498)
>
>We still have some concerns regarding the design when it comes to occlusions. https://github.com/w3ctag/design-reviews/issues/507 Just noticed that it was lumped together.

and @BoCupp-Microsoft reopened the issue with: 

> Reopening this issue based on this comment from @cynthia .



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/w3c/editing/issues/325