Keyboard Focus, Global Shortcuts and LWIM

[viridia]

Keyboard Focus, Global Shortcuts and LWIM

Background

The term "focus" has been used in UI design for a long time, stretching back almost to the beginning of GUIs, although the definition of the term has evolved over the decades. In most desktop UI toolkits, and on the web, the word "focus" specifically refers to "keyboard focus". This represents a state in which one widget has special status, such that it gets first crack at handling keyboard events before any other widget does.

In the context of games, we want to broaden the definition of focus to also include gamepads. Many games, particularly console games, allow for gamepad-based navigation between focusable widgets. (For example, I was just playing Diablo IV and using the gamepad to navigate the inventory screen.)

Focus also includes some means of navigation between widgets using the keyboard. On the web and in desktop apps, widgets are often arranged sequentially, that is, there is a global "tab order". However, in console games, a different approach is typically used, where widgets are arranged in a 2D space and the directional buttons are used for navigation.

Keyboard focus is also essential for accessibility. There are many different screen reader products, with varying features and levels of support, but what they all have in common is that they watch the current focus element.

Tab navigation also requires a visual indicator of which element has focus. However, UI designers have historically considered these indicators to be ugly and distracting, as they can clash with the clean aesthetic of modern UIs. For this reason, most desktop UI frameworks automatically hide the focus indicator until it's needed - the indicator is hidden until the framework detects the user is using tab navigation, at which point the focus indicator is made visible.

The term "focus" is also occasionally used to represent the current hover element, that is, the element that the mouse is hovering over. To avoid confusion, I'll refer to this element as the "hover" element rather than "focus". Some apps, such as Blender, allow keyboard shortcuts to be different when hovering over different panels, and some users of Bevy have expressed a desire for this behavior.

Dispatching keyboard events requires careful consideration. For example, suppose a text entry widget has keyboard focus and the user is typing. Any command keys, such as Ctrl-Z (Undo) or Ctrl-X (Cut) need to be processed by the text widget first, exclusive of any other handler. The text widget can, if it recognizes the command, call stop_propagation() on the event, preventing it from being handled as a global shortcut. Other commands, such as Ctrl-S (Save) can be ignored by the text widget, allowing them to propagate upward and be handled by the global shortcut system.

The goal of this document is to lay out a design which permits keyboard and gamepad focus to co-exist with global shortcut managers such as LWIM and Blender-like contextual shortcuts.

Requirements

• Support for focus navigation on both keyboards and gamepads.
• Support different focus navigation strategies.
• Support for integration with LWIM
• Compatible with Bevy accessibility (bevy_a11y).
• Support accessibility best practices, such as "tab trapping".
• Optionally allow for contextual shortcuts based on hover.

Bubbling Strategy

The strategy for dispatching focus-based keyboard events is fairly simple: we use the existing bevy_a11y::Focus resource, which contains an entity reference of the current focus element. Any keyboard or gamepad input events are triggered on this element, and bubble upward using the standard Bevy event bubbling. Any widget in the ancestor chain can choose to handle these events, preventing other, higher-level ancestors from seeing these events.

So for example, a text input widget, when clicked, would set the Focus resource to point to itself. This would cause any subsequent keyboard events to be routed to the widget for processing. The text input can also decline to handle keyboard events it does not recognize, allowing them to be handled at a higher level.

Simple widgets such as buttons and checkboxes can use the same strategy - the only difference is that they respond to fewer keys. A button will respond to SPACE or ENTER, which is equivalent to a button press. All other events are ignored.

Global Shortcuts

If a key event makes it all the way to the root of the hierarchy without being intercepted by a widget, we want to treat it as a global shortcut.

Unfortunately, Bevy events, like most event-based frameworks, don't provide a way to detect whether or not a key event was "handled".

To get around this limitation, we need to add a top-level handler at the root of the hierarchy, one which scoops up any keys which were left over from the widget handlers. This top-level handler can then use some logic to decide how to interpret the event. This could be LWIM, or it could be just a match statement. It could also use other criteria, such as Bevy game states, to decide how to process the key.

What about the case where there is no focus? The Focus resource might be set to None. On the web, clicking on the background will typically clear the focus, so that no element has focus. The question then becomes, which entity do we dispatch the keyboard event to? We said earlier that the root of the hierarchy has a global keyboard handler, but Bevy allows multiple UI hierarchies - which one will we send the event to?

There are a couple of ways we can solve this. One is by setting up a secondary event dispatch mechanism for cases where there is no focus - that is, we don't use bubbling but instead send the event to the global shortcut handler directly. However, this means implementing two different ways of dispatching events.

A different approach, one which takes advantage of the bubbling strategy, is to add a marker component at the hierarchy root - this marker, which we can call DefaultInputHandler, indicates that keyboard events should be dispatched to this entity if there is no focus set. This allows us to use the same pathways for both the focus and non-focus case.

Integration with LWIM

Users may want to use LWIM as the global shortcut manager. This means that LWIM will be the last handler in the chain.

An alternative approach is to process all events through LWIM first, and then dispatch the processed events via bubbling. However, this is problematic: it means, for example, that all normal key presses for typing text would need to be processed via the LWIM map; it also means that the LWIM map would need to understand all of the possible key commands for every type of widget.

Consider, for example, that a text input widget might interpret an up-arrow to mean "move the cursor to the previous line", whereas a numeric spinbox might interpret it to mean "increment the value by 1". LWIM won't know whether to map up-arrow to a MOVE_PREVIOUS_LINE event or a INCREMENT_VALUE event - the only thing it can do is map it to something generic like UP_ARROW, which means that there's not much value in the mapping function.

Hover-based dispatching

With Blender-style contextual shortcuts, we can dispatch the keyboard event to the current hover element, and have it bubble upward just like we did in the focus case. This means that each panel can have a keyboard handler which understands the shortcuts for that panel, and will receive any events that bubble up from widgets in that panel currently being hovered.

However this creates a contradiction: what happens if we have a text input widget that is focused, but the mouse is hovering over a different panel? Now we have two different potential targets for bubbling - which one do we choose?

We have no way to detect whether a keyboard event has been handled other than by adding a handler for it. So one strategy is to dispatch the event twice: one for the focus entity, and once for the hover entity.

The focus entity gets first crack at the input event. Then, the event bubbles upward until it reaches a handler at the top of the UI hierarchy, at which point the root-level handler re-dispatches the event to the current hover entity. If the focus element stops propagation, then the hover event is not dispatched.

To avoid top-level handlers getting double-events (because the focus widget and the hover widget may have a common parent), we'll use different event types: a FocusKeyboardEvent for events dispatched to the focus entity, and a HoverKeyboardEvent for events dispatches to the hover entity.

Note that in this scheme, there's no longer any need for a DefaultInputHandler marker - because even when there is no focus, the HoverKeyboardEvent is still dispatched. Thus, global shortcuts that listen for HoverKeyboardEvent will always get the event when there is no focus.

Focus Navigation

Unfortunately, there's no general agreement on how focus navigation should work: whether it should be sequential (like desktop and web), spatial, network-based and so on.

The good news is that focus navigation can be handled externally, meaning that third-party authors can implement navigation using whatever algorithm they wish, without the need for Bevy to mandate any officially-sanctioned strategy.

Tab-key navigation is simply one kind of global shortcut, one which modifies the Focus resource. This can be provided by authors of widget libraries. Features such as "tab groups" or "tab trapping" can be provided by adding marker components.

By the same token, authors can implement various kinds of 2D spatial keyboard navigation as add-on libraries. For example, you could either use the 2D screen coordinates to search for a tabbable entity in a particular direction, or define components that link elements in a directional network.

Authors can also implement focus hiding - a global resource such as FocusVisibility can be used to indicate whether the current focus indicator should be shown, and individual widgets can implement outlines or whatever strategy they want for displaying focus.

[alice-i-cecile]

This is an excellent summary of the motivation and a solid design. I'm fully on board. IMO the concept of "focus" should be native to bevy itself, and bevy_a11y should be an optional dependency that communicates this back-and-forth with accesskit.

[viridia]

Sure, I'm fine with having Focus be separate from a11y. Perhaps we should name it KeyboardFocus to avoid confusion. We should also perhaps include focus visibility, either as a separate resource, or as a field within KeyboardFocus. It will be up to the tab navigation framework to decide when visibility should be enabled.

[alice-i-cecile]

For keyboard navigation, I'd like us to learn from bevy_picking, and present an API that crates can use to provide "navigation backends". We should ship with at least one, and can add more over time.

[viridia]

My thinking on this is that the standard widget library for the editor will include a tab navigation system, which can be re-used by other people who want a similar (desktop-like) experience. It will of course be it's own crate so people can use it even if they want to use a different widget library. I've already written an implementation of this, but it will need to be ported to whatever focus API we come up with.

[viridia]

@alice-i-cecile There are some open questions about integration with LWIM:

• How hard will it be to configure LWIM so that it can be fed events from the bubbled observers?
• For hover-based conceptual shortcuts, I can envision use cases where users might want to define the per-panel shortcuts using LWIM as well. This would allow a "key map editor" that works for both global and contextual shortcuts. This would require some way to inform LWIM about which panel the events are coming from, so it can distinguish the different contexts. Perhaps the keyboard event handler on the panel could modify the event in some way and then let it bubble to the top.

[alice-i-cecile]

Generally should be quite easy. I would expect that change to be a single PR.

For the second point, I think that having action types and input maps being bound to the UI entity is likely to be the way to go.

[s-puig]

This might not be the place, but wouldn't that be solved by making LWIM stateful?

I originally wanted to make an input manager using computed states to enable action layering/sets. Didn't get very far, but should be possible in theory.

It seems like the same sort of problem.

[alice-i-cecile]

Yeah, LWIM has very good disabling functionality at this point that can be easily hooked into various schemes like this :)

[UkoeHB]

To get around this limitation, we need to add a top-level handler at the root of the hierarchy, one which scoops up any keys which were left over from the widget handlers. This top-level handler can then use some logic to decide how to interpret the event.

This got me thinking, and there's a nice API I can build in bevy_cobweb_ui. Right now I use 'entity events' for interactions, which send an event 'at' a specific entity (similar to an observer). You can register any number of reactors that run when an event is sent at a specific entity. Specific reactors use the EntityEvent<E> system param to read event data, and general reactors use AnyEntityEvent<E>.

I can add a flag 'should bubble' that can be set from EntityEvent or AnyEntityEvent by any of the reactors. Then when all reactors are finished with an entity event, if the flag is set then look up the hierarchy from the target entity for any entities with a registered reactor for that event type. If one is found, send another entity event at that entity (can reuse the original event data), and reactors on that entity can choose to set 'should bubble' again (you could use a shared AnyEntityEvent reactor to check for a marker component like 'ShouldBubble' and then set the flag automatically; user-land reactors could then just handle business logic). If no ancestor is found, broadcast an Unhandled<E> event and use BroadcastEvent<Unhandled<E>> system param in other reactors to deal with it (and Unhandled can collect info about the event like original targeted entity, last-seen entity, number of targeted entities in the bubble chain).

• This would solve your question about global handlers: let the event bubbling terminate in Unhandled<E> events, and use Unhandled<E> if Focus is None.

Anyway, still reading your post :)

[UkoeHB]

The focus entity gets first crack at the input event. Then, the event bubbles upward until it reaches a handler at the top of the UI hierarchy, at which point the root-level handler re-dispatches the event to the current hover entity. If the focus element stops propagation, then the hover event is not dispatched.

I like this idea. The 'global handler' can check if the event targets the Focus entity, and then redispatch to the hover entity if different. And then when the hover fails and that handler runs again, use the global fallback.

To avoid top-level handlers getting double-events (because the focus widget and the hover widget may have a common parent), we'll use different event types:

I agree separate events would be best. It eliminates ambiguity from the user's point of view about under what conditions an event will show up.

[viridia]

I've decided that I don't care for the name HoverKeyboardEvent. Bikeshedding welcome; some ideas are:

• ShortcutKeyboardEvent
• CommandKeyboardEvent

I think FocusKeyboardEvent is probably fine.

[viridia]

I realized today that there's an alternate strategy for doing hover-based shortcuts: simply have an Enter and Leave handler for the various panels, and adjust the global shortcut map when these events are received. This eliminates the need for the double dispatch and two different key event types.

It also avoids a nasty degenerate case: where to dispatch the event to when the mouse is outside the window. And it also decouples the contextual shortcut strategy from the rest of the keyboard dispatching system, allowing it to be an add-on (not everyone is going to want this behavior so it should be optional).

We could make this even simpler if there was a way to define a "catch-all" observer that could receive any events that bubbled up to the top of the hierarchy without anyone calling propagate(false) on the event. This would allow the catch-all to be initialized one time (most likely in a plugin) instead of having to place the DefaultKeyHandler marker on the UI root.

[UkoeHB]

We could make this even simpler if there was a way to define a "catch-all" observer that could receive any events that bubbled up to the top of the hierarchy without anyone calling propagate(false) on the event. This would allow the catch-all to be initialized one time (most likely in a plugin) instead of having to place the DefaultKeyHandler marker on the UI root.

Right, this is why I think the bubbling algorithm should automatically dispatch an 'unhandled' event when bubbling fails to be consumed.

[Shatur]

A bit late to the party, but today I released a plugin for contextual input mappings, and I'm also very interested in integrating it with the upcoming focus API.

Both LWIM and my crate currently read pressed events from resources (like ButtonInput<KeyCode>) rather than using an event-based approach. If you provide an event-based API for consumed inputs, we could collect them and check for consumption when evaluating actions. However, it would be more convenient to have a centralized API for declaring and reading consumed inputs 🤔