Virtual Keyboard API - boundaries of docked overlay keyboard

[kenchris]

Hello TAG!

I'm requesting a TAG review of VirtualKeyboard API .

The Virtual Keyboard (VK) is the on-screen keyboard used for input in scenarios where a hardware keyboard may not be available. Unlike a hardware keyboard, a VK can adapt its shape to optimize for the expected type of input. Authors have control over the displayed shape of the VK through the inputmode attribute, but have limited control over when the VK is shown or hidden.
We propose a new VirtualKeyboard interface that has APIs to provide authors with more control over when the VK is shown or hidden and also exposes the size of the VK so the page can reflow its content if part of it is occluded by the VK.

• Explainer (minimally containing user needs and example code): https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardAPI/explainer.md
• Security and Privacy self-review²: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardPolicy/security-privacy.md
• GitHub repo (if you prefer feedback filed there): https://github.com/MicrosoftEdge/MSEdgeExplainers/issues
• Primary contacts (and their relationship to the specification):
  • Anupam Snigdha (@snianu), Microsoft
  • Bo Cupp(@BoCupp-Microsoft), Microsoft
  • Grisha Lyukshin(@gked), Microsoft
• Organization/project driving the design: Microsoft
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status):
  https://www.chromestatus.com/feature/5680057076940800

Further details:

• [X ] I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): Editing Task Force
• The group where standardization of this work is intended to be done ("unknown" if not known):
  Editing Task Force
• Existing major pieces of multi-stakeholder review or discussion of this design: inputPanelPolicy as a way to control software keyboard w3c/editing#225
• Major unresolved issues with or opposition to this design: N/A
• This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

[kenchris]

I am positive over the new overlaysContent + inset idea (I have suggested similar myself before).

As using an inset doesn't require JavaScript, I am wondering whether there will be another way to set this in addition to a JavaScript only API?

"overlaygeometrychange" sounds quite generic, and as there can be other overlays (other application windows, modal dialogs, file pickers etc) that might be a bit confusing. Are the values flipped (or should they be) in a right-to-left layout?

The boundingRect is currently on the event itself, but it might make sense to additionally also have it on the virtualKeyboard object like the generic sensors do. Or maybe only have it there (@anssiko)

[BoCupp-Microsoft]

Are the values flipped (or should they be) in a right-to-left layout?

No they are not flipped. They are in client coordinates. I don't think they should be flipped as the location of the keyboard is not writing mode dependent.

[snianu]

As using an inset doesn't require JavaScript, I am wondering whether there will be another way to set this in addition to a JavaScript only API?

Do you mean the virtualKeyboard geometry? You do have to register for VK updates so there has to be JS involved anyways, but if there is a way to do this without JS, then I would love to hear it.

"overlaygeometrychange" sounds quite generic, and as there can be other overlays

We are open to suggestions on naming this event.

The boundingRect is currently on the event itself, but it might make sense to additionally also have it on the virtualKeyboard object like the generic sensors

This sounds good to me. I'll open an issue to make this change.

[kenchris]

Do you mean the virtualKeyboard geometry? You do have to register for VK updates so there has to be JS involved anyways, but if there is a way to do this without JS, then I would love to hear it.

The keyboard is docked so it should be possible to represent its position using CSS environmental variables, just like the window segments are with the CSS spanning proposal from Microsoft

[kenchris]

We are open to suggestions on naming this event.

If it only affect the virtual keyboard we could just call it virtualkeyboardgeometrychange - or just geometrychange if it is an even on the virtual keyboard object

[snianu]

The keyboard is docked so it should be possible to represent its position using CSS environmental variables, just like the window segments are with the CSS spanning proposal from Microsoft

Yes, this is definitely we can explore more and incorporate in VirtualKeyboard API. Thanks for clarifying.

If it only affect the virtual keyboard we could just call it virtualkeyboardgeometrychange - or just geometrychange if it is an even on the virtual keyboard object

geometrychange sounds good to me as this event is fired on the virtualkeyboard object.

[cynthia]

Simple question - does this potentially expose extra bits of entropy for fingerprinting over what is already out there? (window resizing)

[BoCupp-Microsoft]

@cynthia there's no new information to be queried that reveals any aspect of the user's machine or its configuration. I suppose the boundingRect exposed in the geometrychanged event will provide a width for the keyboard instead of just the height (which could already be inferred from APIs like Visual Viewport). That information is only available when the user interacts with the page in a way that requires text input. I don't think there is any new privacy concern with this API.

[alice]

Possibly silly question: if the point of geometrychange is simply to allow restyling certain elements in the page to accommodate the keyboard, would it be possible to provide the necessary constraints via CSS environment variables or similar, rather than fire a whole event?

e.g.

.search-box {
   bottom: env(safe-keyboard-inset);
}

I see there is some discussion above about an inset - did something change in this area?

What other actions might an author take in response to that event?

[cynthia]

So, @alice and I took a look at this during our VF2F and there is a huge meta-question about this in general - aside from the declarative bit noted by Alice above.

The proposal notifies web applications when something (in this case, a keyboard) obstructs the content, and communicates that through two scalar values (width/height). The meta-question is - this is simply something that defines content obstruction through a non-content entity, so why is namespaced in the virtual keyboard? There are multiple cases where such things can happen (e.g. picture-in-picture, browser chrome dialogs, etc) which could benefit from communicating this information back to the content.

Further questions:

1. The geometry model assumes that the origin is fixed, and only has a single obstruction. How would it work for something like this? https://help.apple.com/assets/5E989062094622C539A9FD75/5E989065094622C539A9FD8A/en_US/d92415de0336bf743eb7c4b6d4540999.png
2. Were multiple obstructions considered but insignificant or was that missed in the design?
3. What would be a better namespace, if this is no longer scoped to virtual keyboards?

I might be digging the rabbit hole deeper than necessary, but I would like to hear your thoughts on this.

[cynthia]

@cynthia there's no new information to be queried that reveals any aspect of the user's machine or its configuration.

@BoCupp-Microsoft thank you for the clarification.

[BoCupp-Microsoft]

Hi @alice and @cynthia,

Thanks for your questions.

@alice, we're in support of using css environment variables but haven't written a proposal for it yet. @snianu, could you make an update to the explainer? Note we think the geometrychange event is necessary even if we have them since an author could want to scroll an element into view, which can't be done with CSS alone.

@cynthia regarding your meta-question, we don't believe that authors want to respond in the same way to all content obstructions. In the [explainer] (https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/VirtualKeyboardAPI/explainer.md) there are examples that position a searchbox above the keyboard. The searchbox should likely not be repositioned above a browser dialog or picture-in-picture UI. For that reason we want to explicitly describe changes in the visibility and position and size of the virtual keyboard.

Responses to your further questions below:

Regarding point 1, we propose that the author's ability to adjust the site in response to geometry changes for the virtual keyboard is limited to docked virtual keyboards. That prevents the site from trying to reposition its content while the user is trying to explicitly position the keyboard. In the image you referenced, I believe that is a floating keyboard. I don't think there's a docked version of it, is there?

Regarding point 2, related to the answer above, we don't have any known cases where docked keyboards result in multiple obstructions.

I do realize both of my answers to points 1 and 2 are basically dodging the question by claiming there aren't any current cases that require considering multiple obstructions. If we do decide we need to develop for that case, I think we could define the existing boundingRect property of the geometrychange event to be the rectangle that bounds all parts, and define an additional sequence of rects representing each part individually. I would expect sites developed against the current proposed API shape to behave well with a future docked, split keyboard, as there's no requirement to flow content through a split keyboard. And this future API extension would allow sites that want to be docked, split keyboard-aware to do something more complicated with the new rect sequence. So in short, we haven't designed for this case, but we see a clear path to solving that case if the need arises.

Regarding point 3, I think my answer to the meta question shows that we don't expect there to be one generalized API to handle all obstructions. If that was the approach we wanted to take then there is a Visual Viewport API which would be the best candidate.

Note the Virtual Keyboard API's relation to the the Visual Viewport API has been discussed here. I think at least @othermaciej feels differently, but my current thinking is that the API surface of the VirtualKeyboard interface is not limited to just describing the rectangle of how the feature intersects the content: there are hide and show APIs and a property that indicates whether the author has opted out of having the Visual Viewport resize when the virtual keyboard is shown. IMO its more natural to group these APIs together, under a namespace related to their common purpose, instead of putting APIs to control visibility on one interface, and events that generically describe change to the viewport geometry (that may not relate to the appearance of the virtual keyboard) on the Visual Viewport interface.

Hope that helps.

[kenchris]

Extending this to other types of occlusions might make sense, but I don't think we should extend it to any occlusion.

I understand the use-cases behind virtual keyboard and picture-in-picture, but I am afraid that we will suddenly start to expose occlusions like "web bluetooth selector modal dialog" without actually having use-cases for adoption to that.

Also if we go all aboard and support occlusions by other OS windows, this might even add finger-printability, by knowing the default size of say a Skype window, that will pop on top when I to alt-tab cycling. Or knowing that there is a floating Facebook messenger head on top.

The only real use-case I can come up with for supporting generic occlusions is to modify the position of Ads to always be visible :-) which I am sure is a terrible idea that some surely will pursue.

That said, I am find with supporting keyboard and pip which are related to the web page itself, but they developer needs to know what occlusion belongs to what region to which of these.

[cynthia]

Extending this to other types of occlusions might make sense, but I don't think we should extend it to any occlusion.

I don't think we implied that above, the intent was being able to find all occlusions in one place instead of a bajillion different places. The intent is to provide a shared repository of occlusions when a new spec needs to add one, not to add all existing occlusions into this.

[cynthia]

@snianu @BoCupp-Microsoft @gked Is this still on-going work? We haven't heard back from you and we noticed that there might have been a misunderstanding - we closed and considered the other VK API issue as good to go, but we think there still is room for improvement here.

The response here MicrosoftEdge/MSEdgeExplainers#389 starts with:

The VisualViewport only solves the problem of knowing when the visual viewport is changing. It is not a general purpose reporter of occlusions and provides no information today about occlusions

But what we see is the actual occlusion still inside the virtualKeyboard object. (which does not make it general purpose, since it's impossible to extend unless we implement a common interface that enforces boundingRect. With that it's still unclear how the user is supposed to enumerate these occlusions.) I think it's fine that there is a pointer to it from the object, but we still think that there should be a canonical place to look for when it comes to occlusions, and we don't see that addressed. @alice and I discussed this today, and was wondering if this functionality could be added to the Visual Viewport API. (even if it has shipped, that doesn't mean adding features is impossible.)

[BoCupp-Microsoft]

Yes, the work continues. We have an implementation behind a flag in Chromium browsers which matches our explainer and will turn our explainer into a spec sometime soon.

I reopened this issue (MicrosoftEdge/MSEdgeExplainers#389) for now while our discussion continues. Apologies for closing it prematurely.

But what we see is the actual occlusion still inside the virtualKeyboard object.

Yes. We describe how a docked, overlaid virtual keyboard object occludes the layout viewport via the VirtualKeyboard interface.

I made an argument in the issue linked above as to why the virtual keyboard seems like a special case when compared with other examples like picture-in-picture or windows that overlap with the browser in a desktop OS. Here's the relevant bit:

In general I don't think we want to report all the things layered over top of the visual viewport. I wouldn't want web pages updating their layout as I ALT+TAB to another application that overlaps my browser, or have a web page fight me as a drag around a floating keyboard or a picture-in-picture window.

The "docked" virtual keyboard is a special case. When it appears, users do expect to be able to see what they're typing, but authors may not want the experience we offer by default. L-shaped viewports aside, adjusting the visual viewport means we'll be creating the "slop" I describe above and jumping part of the page out of the visual viewport to ensure the user can see the blinking caret. If I'm an author that has created a web application whose UI fits in 100vw x 100vh, I might rather adjust the height of one of my subscrollers rather than "unglue" my UI so that the user can start panning it around.

So with regards to this comment:

it's still unclear how the user is supposed to enumerate these occlusions.

My answer is that I don't want authors to be able to enumerate other arbitrary occlusions. If there are some specific occlusions that make sense, then I'd like to consider those when we have the specifics. I'm not aware of any customers asking to relayout their UI while other application windows are being moved around or the user moves the floating keyboard or the user repositions picture-in-picture. If there were such requests, I would still be skeptical whether exposing that information would really enable better user experiences.

[snianu]

@cynthia Could you please let us know if the above answers address your concerns or not? This is currently blocking spec review & shipping in Chromium as based on the outcome of this discussion it may lead to significant changes in the API structure/code. We had this discussion at the TPAC meeting and we came to a consensus that VisualViewport is not the right place to fire geometrychange event caused by virtual keyboard show/hide. One of the reasons is with overlaysContent flag set, the VK doesn't change the VisualViewport i.e. the keyboard is overlaid on top of the content and doesn't "resize" or change the VisualViewport in any way. These discussions were buried in the show/hide policy thread which was opened separately. The discussions about VisualViewport APIs start from here. The MoM of the TPAC breakout session discussion is here
Thank you for all the feedback so far!

[kenchris]

This is my personal take as there is not full consensus in the TAG at this point.

I believe virtual keyboards are special enough to warrant a specific API instead of a generic occlusion API. If such a generic API could be created it would be perfect, but I am wondering if there are strong enough use-cases to drive the development of such an API. Example occlusions:

• Other windows
• Picture in picture
• Emoji picker
• Virtual keyboards (even split ones)

I am happy that you included CSS environmental variables as that makes it much easier to use this feature from CSS.

I am also fine with this not being integrated with the Virtual Viewport API given its state of development.

I am a little bit concerned about navigator.virtualKeyboard.overlaysContent as it might not be clear that it is a setter, but that is nitpicking. I am more concerned that I as a developer can set it to false, but if the user is using a split keyboard, it is going to overlay content anyways. Maybe it should be an enum like preferredMode: overlay | docked.

As overlay keyboards are used on existing platforms, supporting them out of the box seems quite important and that is also what I am hearing from @cynthia - This requires the supporting at least two bounding boxes. This also requires additions to the CSS support @atanassov

[cynthia]

Hi, apologies for the delayed response - this fell through the cracks (never got a proper milestone update - our bad) and luckily was noticed by @plinss.

Could you please let us know if the above answers address your concerns or not?

To be honest, I'm afraid not - but given this:

This is currently blocking spec review & shipping in Chromium as based on the outcome of this discussion it may lead to significant changes in the API structure/code.

We also don't think we should block work - the core concerns have been raised long enough ago, but we don't think it was addressed in a timely manner. However, given that there is a deadline and we don't think there is much else for us to do here.

That said, we have also noticed that representatives from different implementors have raised [1] [2] similar concerns and we do not think the concerns have been properly addressed. For these reasons, we plan to close this during our plenary.

[1] w3c/editing#225 (comment)
[2] w3c/editing#225 (comment)

[torgo]

Hi folks – thanks again for sending this review request. We are going to close this with an 'ambivalent' status. As indicated above, some of the feedback has been actioned but we feel there are some important points that haven't been addressed. More importantly there is an issue with multi-implementer support and considering the work is going in in webapps, you will need to gain implementer support before advancing this spec to the next stage there. We hope that happens.

[jspurlin]

...I'm a little late to the party on this, but I want to share my thoughts...

As a web app developer, the virtual keyboard and it occluding content is unique/specific enough to warrant the functionality to be scoped to its own API, and navigator.virtualKeyboard/navigator.virtualKeyboard.overlaysContent makes sense for implementing associated scenarios in a web application.

In my mind, attempting to lump the functionality into with Visual Viewport does nothing but over complicate the the mental model for how to think about and approach scenarios related to virtual keyboard (which are very targeted and different from any other Visual Viewport interactions). Additionally, I think we also need be mindful of not trying to come up with a "kitchen sink" solution that solves for unrelated scenarios that don't exist today... doing this could lead to an over-engineered solution that ends up being too complicated for practical purposes.

As it relates to occluding content, I feel the virtualKeyboard API should encapsulate allowing web developers to have knowledge about its presence/dimensions (overlaygeometrychange and boundingRect) and be able to tell the browser to automatically occlude content with the virtual keyboard (overlaysContent).

I'm happy to discuss this in more detail.