inputPanelPolicy as a way to control software keyboard

[gked]

It has been brought up by few online editors that they need a way to control a software keyboard by not making it to show up in certain cases. Detailed discussion can be found here.

EditContext API proposal exposes such an attribute and we think, it would be beneficial to expose it on all editable elements.

The idea is to introduce an inputPanelPolicy attribute that when set to manual will prevent software keyboard from showing up even when there is user focus tap. This is a html spec related issue, but we wanted to get some feedback from the community on the proposed solution.

some early thinking:

inputPanelPolicy when set to manual before user taps on the editable field, will prevent the software keyboard from showing up.

If a web developer wants to drive software keyboard regardless of its current state, they could achieve that with also using additional inputPanelPolicy attribute values. Something like hidden and show.

So there could be a total of four attribute values: [auto, manual, show, hidden] to controls behavior of and also to drive the keyboard to appear or get dismissed.

[EDIT] I've removed reference to inputMode as it is really about controlling the layout of the virtual keyboard.

[gked]

Resolution call 2020-01-10: @gked will create a PR to HTML spec with the new attribute proposal.

[othermaciej]

Why the name inputPanelPolicy when what it controls is the "software keyboard"? Why not a more obvious name?

Also most web APIs say auto, not automatic.

[gked]

Why the name inputPanelPolicy when what it controls is the "software keyboard"? Why not a more obvious name?

@othermaciej thank you for your feedback. Well, inputPanel is more than just a keyboard, it could also be an inking panel:

Another scenario I could think of is some Dictation UI powered by the OS.

Also most web APIs say auto, not automatic.

Fair point I will update the post.

[othermaciej]

At least on iOS, handwriting keyboards are still considered keyboards. And on any platform/browser that allowed inline handwriting speech entry, I would not expect this API to suppress it. Beyond the mobile touch UI model, I'm not sure this API actually generalizes.

(I'm not clear enough on the intended use case for suppressing dictation UI; I haven't personally seen elaborate UI like that.)

And in any case, I think naming after the most obvious/common use case helps in API just as it does in issue titles. After all, this issue is not titled with "a way to control input panel" b/c it would't be obvious what that means.

One other terminology note: "onscreen keyboard" seems more precise than "software keyboard", even if it's just a description and not in the API name. There can be such a thing as a keyboard totally driven by software which is not on the same screen as the content and which therefore should not be suppressed.

[gked]

And in any case, I think naming after the most obvious/common use case helps in API just as it does in issue titles. After all, this issue is not titled with "a way to control input panel" b/c it would't be obvious what that means.

One other terminology note: "onscreen keyboard" seems more precise than "software keyboard", even if it's just a description and not in the API name. There can be such a thing as a keyboard totally driven by software which is not on the same screen as the content and which therefore should not be suppressed.

Agreed and I am not attached to the name perse. I would be opened to other suggestions.
We can iterate on the name/shape of the api once there is a PR on html spec. Either me or @snianu
will create it in a week or so against html spec.

[gked]

Spec draft - https://docs.google.com/document/d/1CBeWoGxQBpGLXSD2DwfVBf7hmS8Rmxb--Mu9F3Bj24o/edit?usp=sharing

[othermaciej]

@gked Thanks for writing this up!

Could you format that as an Explainer on GitHub (perhaps in this group's space)?

(As it stands, it's more of an Explainer-level design document than a spec draft, and in any case we don't usually use Google Docs either for explainers or for specs).

[gked]

@BoCupp-Microsoft and @snianu to give credit to for this
Yes, we use the same process. Sorry for the imprecise terminology. Definitely not a spec document.
We'll convert it to the explainer and publish it on Microsoft explainers page.

[gked]

@othermaciej FYI - here is the public explainer

[snianu]

@rniwa and @domenic Could you please give us some feedback on this proposal. This would not only address the dynamic VK layout changes, but would also let the script authors manage VK behavior through explicit show/hide APIs.

[domenic]

I'm probably not the right person from Chromium here to look at virtual keyboard things; paging @mfreed7 or @chrishtr.

[chrishtr]

@NavidZ

[rniwa]

Given there is no mechanism to manually bring up a software keyboard on iOS, I don't think this API makes much sense to support on iOS.

[snianu]

@rniwa What do you mean by "manually bring up a software keyboard" exactly? Like a button in the shell that pulls up the VK or a system API that can be called to show the VK? I think there is a similar API on Android(and maybe ChromeOS?) that lets you raise the VK through a system API call, but yes, pulling up the virtual keyboard through a button in the shell is I guess only supported on Windows for now. The API proposal is not just for scenarios where user can pull up the VK through a button in the shell. It is basically a mechanism to control VK behavior through system API calls which currently can't be done deterministically in JS.

[whsieh]

So the issue is that when virtualkeyboardpolicy=“manual” is specified on iOS, the user would have no way of bringing up the keyboard because there is no UI (e.g. a button, or gesture) that would allow them to show the keyboard while the element is focused.

It would need to be solely up to the web developer to use the VirtualKeyboard.show() API to summon the keyboard.

[snianu]

Well, if there is no system API to raise/dismiss VK, then I'm guessing there wouldn't be an API to suppress VK either? It would always behave like virtualkeyboardpolicy=“auto” as the focus is inside an editable element. virtualkeyboardpolicy=“manual” behavior in editable elements can only be achieved if there is a way for the platform to suppress VK and call a system API explicitly to show/hide VK.
But if there is an API to control the VK behavior and web authors used virtualkeyboardpolicy=“manual” and didn't call show()/hide() when the editable element is focused, then I think it should be considered as a site issue. Currently web authors can do this by using inputMode="none" and never change the value of it when the user sets focus into the editable element.

[whsieh]

Ah, so there does exist system API on iOS to show or dismiss the keyboard at arbitrary times. Ryosuke and I were more worried about the user not being able to interact with UI in order to bring up the keyboard in the case where the website doesn’t call show().

You’re right that this would be a site bug, but at the same time, we’re concerned that web authors may only test on devices that support a manual “show keyboard” button (e.g. Microsoft Surface), and not realize that adding only this attribute with no accompanying logic to show the keyboard will cause issues for iOS users.

[snianu]

That is a good point. I think we somehow missed to address this part in the explainer, but we did discuss about it when we were reviewing this API with our OS team.
The user may not want some sites to have the ability to show or dismiss their VK. These sites may be malicious and attempt to DOS the user by denying them access to their VK. Alternatively, the site may simply be poorly implemented and the user would prefer to lock the site into auto behavior rather than have their VK be mismanaged.
To prevent DOS attacks and combat poor experiences created by authors that improperly use the APIs, a site must receive permission to explicitly control the VK using the show and hide APIs. Additionally, using the value manual for the virtualKeyboardPolicy is equivalent to auto unless the site has received permission to control the VK. The rationale is that manual effectively disables automatic management of the VK, and if the site hasn’t received permission to explicitly control it then the keyboard may become inaccessible to the user.
The proposed permission name is “virtualkeyboard”.
Permission should be requested when the show or hide methods are called, or when an editable element with a virtualkeyboardpolicy of manual is focused or tapped.

Additionally, calls to show or hide the keyboard should only be honored (or lead to a permission prompt) when they are received from a realm with an active document which has received some user interaction.
Thoughts?

[snianu]

@smaug---- Could you please give us some feedback or tag the appropriate person from Firefox who could review this proposal?

[rniwa]

The user may not want some sites to have the ability to show or dismiss their VK. These sites may be malicious and attempt to DOS the user by denying them access to their VK. Alternatively, the site may simply be poorly implemented and the user would prefer to lock the site into auto behavior rather than have their VK be mismanaged.

I don't think malicious case is interesting because it can only affect the website itself. A more common scenario is the lack of knowledge / testing that results in a website being non-functional on iOS devices. Given the number of websites that lock themselves to only support Chrome in wild, I don't think we want to take that kind of risk.

To prevent DOS attacks and combat poor experiences created by authors that improperly use the APIs, a site must receive permission to explicitly control the VK using the show and hide APIs. Additionally, using the value manual for the virtualKeyboardPolicy is equivalent to auto unless the site has received permission to control the VK. The rationale is that manual effectively disables automatic management of the VK, and if the site hasn’t received permission to explicitly control it then the keyboard may become inaccessible to the user.
The proposed permission name is “virtualkeyboard”.
Permission should be requested when the show or hide methods are called, or when an editable element with a virtualkeyboardpolicy of manual is focused or tapped.

I don't think permission makes much sense for this feature since it has such small end-user impact when it works, and the end user are unlikely to fully understand the implication of the issue otherwise.

[NavidZ]

@mustaqahmed from Chrome side.

[snianu]

I think there is some confusion here. Show/Hide APIs that is on VirtualKeyboard interface, is just a way to explicitly trigger VK. Calling show/hide when virtualKeyboardPolicy="manual" is not required to raise/dismiss VK. Web authors could also switch from virtualKeyboardPolicy="manual" to virtualKeyboardPolicy="auto" dynamically without explicitly calling Show/Hide APIs if they wanted the VK to appear on maybe next tap on the edit control when the initial state of the VK was hidden. We propose Show/Hide APIs to let authors have some explicit control over the VK behavior when they want to handle more complicated scenarios (described in the explainer) that couldn't be achieved without these APIs.
Currently inputMode="none" is used to suppress VK, but we investigated that this attribute value doesn't cover all the scenarios that are needed by the sites that require more than
just suppressing the VK on the initial tap. Moreover, inputMode controls the layout of the VK and doesn't provide an explicit control of the VK without affecting the layout
(This is what we were trying to convey in these discussions about changing the definition of what this value does, but I guess we decided on keeping it as it has compat problems).
There are also cases where the sites need to know the geometry of the VK in order to properly layout its contents when parts of it are occluded by the VK. The VirtualKeyboard interface has APIs that address this issue too.

Re permissions: VirtualKeyboard is a feature provided by the OS (Text Input service on Windows to be more precise) and not the Browser. If the VK doesn't work on a platform, then
the user most likely would blame the OS and not the Browser (as seen from user studies that we've done internally). This is why we provided an option to let users have a say as to
when the VK should be controlled by the site vs the browser default behavior. If the site is malfunctioning due to the policies that affect the VK behavior, then at least users have an
option to deny the VK permissions and let Browser control the VK and ignore the policies mentioned by the site. This is just an option that we thought would be helpful to address
some concerns regarding abuse of these APIs by the web authors.

[rniwa]

I think there is some confusion here. Show/Hide APIs that is on VirtualKeyboard interface, is just a way to explicitly trigger VK. Calling show/hide when virtualKeyboardPolicy="manual" is not required to raise/dismiss VK.

There is no confusion there.

Web authors could also switch from virtualKeyboardPolicy="manual" to virtualKeyboardPolicy="auto" dynamically without explicitly calling Show/Hide APIs if they wanted the VK to appear on maybe next tap on the edit control when the initial state of the VK was hidden.

Nor here.

We propose Show/Hide APIs to let authors have some explicit control over the VK behavior when they want to handle more complicated scenarios (described in the explainer) that couldn't be achieved without these APIs.

Currently inputMode="none" is used to suppress VK, but we investigated that this attribute value doesn't cover all the scenarios that are needed by the sites that require more than
just suppressing the VK on the initial tap.

I understand that's a scenario important on your platform (e.g. Surface), but that's wholly adequate on iOS and iPadOS. And what you're proposing isn't not gonna work on our platform because there is simply not a mechanism for user to control when a virtual keyboard will be shown, and override what a website is doing.

Re permissions: VirtualKeyboard is a feature provided by the OS (Text Input service on Windows to be more precise) and not the Browser. If the VK doesn't work on a platform, then
the user most likely would blame the OS and not the Browser (as seen from user studies that we've done internally).

This is problematic for us either way since we make both the browser and OS. In general, we don't want user experience to degrade because there was an oversight by an author.

This is why we provided an option to let users have a say as to
when the VK should be controlled by the site vs the browser default behavior. If the site is malfunctioning due to the policies that affect the VK behavior, then at least users have an
option to deny the VK permissions and let Browser control the VK and ignore the policies mentioned by the site. This is just an option that we thought would be helpful to address
some concerns regarding abuse of these APIs by the web authors.

Again, we don't think permission is the right model for this. Users aren't going to understand what it means, and in cases where it works, it fatigues users of permission dialogs / sheets.

[mustaqahmed]

What if on platforms without a VK button, the browser provides a workaround for users and informs them when inputPanelPolicy=manual? Browsers already do a very similar thing with fullscreen and pointer-lock: they show an "info bar" (is there a name for it?) to remind users to hit "esc" to exit fullscreen/pointer-lock. Since pressing "Esc" key won't work for "manual-mode" VKs, there could be a browser-specific hot-key to bring up the keyboard, or even a keyboard icon on address bar that would flash once.

The triggering of the "info bar" could be tied to click handlers. One possibility is that the browser would show the bar only when JS doesn't call show() in the handler. Or may be when the click handler cancels the event (event.preventDefault()).

Does it look like a reasonable solution?

[BoCupp-Microsoft]

@rniwa @whsieh it sounds like your concern is that authors won't show the VK so that users won't be able to type on iOS. Don't authors already have this power via inputMode="none"? How did you mitigate these concerns?

[othermaciej]

We are not worried about a website doing this maliciously. Then the website just sucks, and no one uses it.

We are worried about a website doing this accidentally, because they only test on other platforms that always have a way to manually bring up the keyboard, and not on iOS/iPadOS, which do not have such a button.

inputMode=none doesn’t seem the same issue because it says to never show the keyboard, not to put it under manual control.

Permissions don’t address this because the permission is not likely to be comprehensible and it will just contribute to alert fatigue.

Adding a browser-specific button to show the keyboard seems like it is fighting the OS design of not having such a button. Users would be puzzled why this is a thing only in the browser. It’s basically changing the design of iOS to match other platforms that have manual control of the keyboard.

This may well be a solvable problem but none of solutions proposed here seem workable so far.

[BoCupp-Microsoft]

We are worried about a website doing this accidentally, because they only test on other platforms that always have a way to manually bring up the keyboard, and not on iOS/iPadOS, which do not have such a button.

Is your specific concern that the developer will rely on the shell to let the user show/hide the keyboard and leave the input controls in manual mode because, e.g. the author prefers to not have the keyboard show up automatically?

I don't think this will happen as it would result in a very poor experience on Windows as well. When a user taps on an edit control on a touch device (without a hardware keyboard attached) the user expects the VK to automatically appear - just like on iOS. The touch keyboard button doesn't show up by default. The user must customize the start bar to show it. It isn't a prominent feature of the OS like, e.g. the back button is on Android phones. Author's won't build sites assuming its presence - at least not based on the Windows experience.

[othermaciej]

I hiding the keyboard without showing it produces a near-equally-bad experience everywhere, then it likely doesn't have the specific hazard of accidental misuse. It would be nice if it was more clear that it's a feature for experts who are doing things manually. I could imagine having it controlled by script, since the attribute almost certainly should not be respected when JavaScript is disabled.

Additional note: why does the Virtual Keyboard API need to provide metrics for the size of the keyboard? Visual Viewport API should already be a way to find out how much space is taken up, whether by the keyboard or other things, including events to monitor it. A number of sites already use it in this capacity, to the point that we implemented it last year so that iPad could better support desktop sites. Seems unnecessary to have two ways to do it.)

(BTW, I prefer "onscreen keyboard" to "virtual keyboard" or "software keyboard", it's more clear about what it actually is. You could imagine a keyboard being "virtual" or software-driven in some way without actually ever appearing on the same screen as content, which is the relevant consideration for these types of features.)

[snianu]

@othermaciej Re: Geometry of the VK - This is mainly for devices with dual screen where the VisualViewport change only gives the rectangle that is occluded by the VK, but really the content that is occluded by the VK only covers one screen and not the other. This explainer has screenshots describing the dual screen in greater detail.
Also just to make sure that I understand it correctly, do we agree that iOS would benefit from the VK proposal? Do you have any immediate concerns that are specific to iOS that would affect the experience of the user because of authors manually controlling the VK?

[smaug----]

Shouldn't VisualViewport API be then improved to cover also that use case, report metrics also in dual screen.
Having two APIs for almost the same thing and often working very similarly feels a tad odd.

[BoCupp-Microsoft]

@smaug---- @othermaciej We didn't want to describe the Visual Viewport with a path instead of a rectangle, i.e. let's not create L-shaped viewports.

The Virtual Keyboard API includes a property "overlaysContent" which tells the browser not to resize the Visual Viewport in response to the appearance of a docked VK. Setting this property true opts in to separate keyboard geometry events and stops reducing the size of the Visual Viewport so as not to waste space in dual screen scenarios.

It is specifically because we're not changing the size of the Visual Viewport that we didn't extend the existing API.

[othermaciej]

@BoCupp-Microsoft that seems even more confusing. One of the main purposes of Visual Viewport is for content to avoid the keyboard. Having a different API that essentially "turns off" this functionality of Visual Viewport and provides similar info does not seem like a coherent API story.

[BoCupp-Microsoft]

@othermaciej as I understand it the Visual Viewport API describes the viewport, not the on screen keyboard. Certainly the keyboard can influence the size of the viewport - so can resizing the browser - but in my mind that doesn't make a viewport API an ideal API surface for controlling or describing the the on screen keyboard.

In these two explainers: Virtual Keyboard API, Virtual Keyboard Policy, we describe the capabilities of the virtual keyboard API:

• requesting that the keyboard be shown
• requesting the the keyboard be hidden
• providing events for when the keyboard is shown or hidden
• providing the size of the keyboard (which will not be the full width of the viewport on some devices)
• instructing the keyboard to not automatically resize the viewport

As an author I would be surprised to find all these keyboard related capabilities on a viewport API. The only connection to the viewport is the ability to request that the keyboard not resize the viewport when shown.

[othermaciej]

It totally makes sense for some of those capabilities to be separate. But the viewport-related ones seem misplaced or duplicative. Visual Viewport API is not required to react to browser resizes. Legacy APIs cover that case ok. The only new use cases it enables are adapting to onscreen keyboards and to adapt to auto-hiding browser UI (since that often doesn't trigger resize events).

If we conclude that Visual Viewport API is not good enough for web content to adapt to onscreen keyboards, and can't be enhanced to provide content authors what they need, then perhaps we should deprecate it, or limit its scope to autohiding UI. But I think that would be a bad outcome. I think Visual Viewport API could be enhanced to report individual overlay rects. That way, authors who want to adapt to available screen space won't have to juggle two different APIs. And we won't need to invent still other new APIs for other forms of onscreen overlays, like PIP, or the iPadOS Universal Callout Bar which appears even with a hardware keyboard attached.

In brief, we should inventing a whole new API each time there is major new work to communicate available screen space to web content. That is an anti-pattern. Instead, let's try to enhance existing APIs.

On the other hand, it seems totally reasonable to me for API to explicitly show or hide the keyboard and the like to be in a different place.