Improve keyboard input with Input<ScanCode>

[TheRawMeatball]

What problem does this solve or what need does it fill?

Right now, getting the status of raw keys is difficult. An Input<ScanCode> would make this simpler.

What solution would you like?

Add a Input<ScanCode> resource by default in bevy_input, and change the ScanCode type to be an enum where the keys are represented as the key they correspond to on a standard QWERTY keyboard.

What alternative(s) have you considered?

Do nothing, or do this partially.

Additional context

Started discussion in discord

[inodentry]

Key Codes are layout-dependent. Scan Codes are layout-independent. I think we should improve our keyboard APIs to encourage good practices:

• Scan Codes should be used for game input.
• Key Codes should be used for text input (and maybe display).

So I would suggest the following:

• Create a new ScanCode enum, that is similar to the current KeyCode enum.
  • It should contain enum variants for every key commonly found on typical keyboards.
  • Alphanumeric keys could be named based on QWERTY (as that is what most people are familiar with).
  • It should be trivially convertable to/from an integer.
  • An Other variant could be included for extra keys on weird keyboards, when not covered by any enum variant.
• Change the KeyCode enum to be better suited for text input and more indicative of its layout-centric nature.
  • Keep the variants for for non-alphanumeric keys like Esc, Delete, Return, ...
  • Replace all alphanumeric keys with a single Char(char) variant, containing the unicode character value, as per the keyboard layout of the user.
    • This should make it trivial to implement text input.
    • It would make it easy for users to support non-latin scripts for text input.
    • It would also encourage people to use ScanCode for game inputs, instead of KeyCode.
• Both Input<KeyCode> and Input<ScanCode> resources should be available.
• Replace the scan_code field in the KeyboardInput event type with a ScanCode (it is currently a bare u32).

These changes should result in a much cleaner keyboard input API that encourages good practices and is easy to use correctly.

[cart]

To me this isn't necessarily clear cut. Game input generally cares about the "shape" of the keyboard (ex: WASD movement or the position of the arrow keys). ScanCode is closer to the ground truth of the "shape". But it also isnt the ground truth because we live in a world with different physical keyboard layouts.

By adopting ScanCode as our standard for game inputs, we are essentially saying that non-qwerty-scancode-aligned keyboard owners need to remap all of the defaults a game provides (or have the game provide alternative mappings for each layout they want to support). This feels pretty english-centric to me.

Using ScanCode for input will likely result in better outcomes, but it isn't a complete solution:

• A keyboard input model rust-windowing/winit#753
• https://hacks.mozilla.org/2017/03/internationalize-your-keyboard-controls/
  • The "reference keyboard" / KeyboardEvent.code is the W3C solution to this problem. It forces everyone to code to a QWERTY layout, but at least its consistent.

Semi-related Bevy conversation: #782

[mrec]

As part of this, would Bevy need a portable mechanism to translate a ScanCode to a KeyCode for the user's keyboard and locale? Otherwise I don't know how a game's instructions are supposed to tell the user what keys to press.

EDIT: hmm, this is the same use case as #1995 which is now closed, but as far as I can see #805 only added a ReceivedCharacter event for text input. Maybe I'm missing something, but I don't see how this helps describe a key which the user isn't pressing.

[inodentry]

By adopting ScanCode as our standard for game inputs, we are essentially saying that non-qwerty-scancode-aligned keyboard owners need to remap all of the defaults a game provides (or have the game provide alternative mappings for each layout they want to support). This feels pretty english-centric to me.

Hmm ... what? This seems backwards to me? What is a "non-qwerty-scancode-aligned keyboard"? I thought the scan codes represent the physical keys and are the same on every keyboard?

If you are referring to my suggestion about naming the enum variants after QWERTY, I can see the point, but those are just internal enum values. I suggest using the matching KeyCode for text display to users / as part of UI.

Game inputs should use scan codes, precisely so that they are not "english-centric". That way they will work regardless of whether i switch my keyboard to QWERTY, Dvorak, or Russian. The current KeyCode-based API is english-centric, because it makes it inconvenient to use non-qwerty (or worse, non-latin) layouts.

But it also isnt the ground truth because we live in a world with different physical keyboard layouts.

Of course some people have special nonstandard keyboards (like the various ergonomic keyboards with weird physical arrangement of keys), but those people are going to have to customize/remap their game inputs anyway.

[cart]

Hmm ... what? This seems backwards to me? What is a "non-qwerty-scancode-aligned keyboard"? I thought the scan codes represent the physical keys and are the same on every keyboard?

This is what I was contesting. I understand that scan code is the raw event sent from the keyboard and doesn't directly correlate to a letter (this is the OS's job, which is what produces KeyCode and is virtual-keyboard-layout dependent). I have only ever used "qwerty" keyboards, but I am currently under the impression that international keyboards exist (such as this azerty keyboard) with scan codes that don't line up with the physical positions of qwerty keyboard scancodes. This is confounded by the fact that other azerty keyboards apparently do use qwerty scan codes (and rely on the OS to configure an azerty layout).

I was under the impression that the W3C KeyboardEvent.code spec was designed to combat this by providing abstract non-scancode values. But maybe they also assume scancode positions are constant and they just 1:1 map scancode to new names.

Of course some people have special nonstandard keyboards (like the various ergonomic keyboards with weird physical arrangement of keys), but those people are going to have to customize/remap their game inputs anyway.

If these are truly "non-standard" in terms of "an overwhelming majority of the world uses the same scan code positions", then I'm totally cool with using scancodes because we can't support every fringe thing that exists. I just want to make sure we aren't alienating some significant portion of the world's population.

I'm definitely not an expert here, but I'd like some way to quantify the actual situation. Something like "windows / mac / linux all assume physical scancode positions are stable as defined by some standard X and only 0.1% of the population uses standard Y".

edit: adjusted qwerty scancode link to something that uses hex values for parity with the other example

[Tsudico]

The USB HID spec for keyboards is based off the physical layout of a QWERTY keyboard so I think any enumeration should use the spec's defined values where applicable. [See Keyboard / Keypad page (0x07) on pdf page 82]:
https://usb.org/document-library/hid-usage-tables-122

[mockersf]

Fun news! on macOS, the scan codes provided by winit don't seem to be the real scan codes, but the virtual key codes macOS is using... https://developer.apple.com/documentation/coregraphics/cgkeycode

my "q" key have a scan code of... 0

[inodentry]

Ugh, why does macOS always ... uhm ... "think different"

[mockersf]

I tried running Bevy with rust-windowing/winit#1890 (macOS impl for rust-windowing/winit#753) and events are much better for keyboard 👍

example event:

KeyEvent {
    physical_key: KeyA,
    logical_key: Character(
        "q",
    ),
    text: Some(
        "q",
    ),
    location: Standard,
    state: Pressed,
    repeat: false,
    platform_specific: KeyEventExtra {
        text_with_all_modifiers: Some(
            "q",
        ),
        key_without_modifiers: Character(
            "q",
        ),
    },
}

It seems done for Windows (rust-windowing/winit#1788), in review for macOS (rust-windowing/winit#1890), draft for web (rust-windowing/winit#1888) and under work for Linux.

Unless we are ready to implement something temporary, I think we should wait for this to be done on winit side.

[cart]

Wow that api is so much better. And yeah the scancodes for mac (even in the new impl) are completely out of sync with what I expect them to be. Sounds like the "physical_key" api is the way to go.

Relevant bit from the api redesign issue:

Which is basically what the W3C decided to roll with for their new keyboard api for the web. Unfortunately this is the sort of thing that could drag on for a long time (iirc winit's eventloop 2.0 started midway through 2018 and winit 20.0 dropped at the start of 2020). We should try to help them if possible.

Bevy contributors motivated to make input better (faster) should consider reaching out in this issue: rust-windowing/winit#1806

[maroider]

Unfortunately this is the sort of thing that could drag on for a long time (iirc winit's eventloop 2.0 started midway through 2018 and winit 20.0 dropped at the start of 2020).

Heh, rust-windowing/winit#753 has been open for well over two years now, so I hope we don't end up spending too much time getting this across the finish line.

Any help on this matter would be appreciated. The Linux impl should have a PR up later this week (hopefully), so what would be most helpful is help with code review, preferably from someone familiar with X11 and/or Wayland, as both these platforms (effectively) lack maintainers. I'm sure @ArturKovacs would also appreciate another pair of eyes (and more testing) on the macOS impl.

[mockersf]

and now rust-windowing/winit#1932 is up for anyone on Linux to test 🎉

[janhohenheim]

Are there any updates on this?

[mlp1802]

I'm not sure I understand this discussion fully, I too also need a way to show the user which keys he has selected, in a controls panel. I did this simple thing. Is this bad?

pub fn create_keycode_map() -> HashMap<KeyCode, String> {
    let mut map = HashMap::default();
    map.insert(KeyCode::A, "A");
    map.insert(KeyCode::B, "B");
    map.insert(KeyCode::C, "C");
    map.insert(KeyCode::D, "D");
    map.insert(KeyCode::E, "E");
    map.insert(KeyCode::F, "F");
    map.insert(KeyCode::G, "G");
    map.insert(KeyCode::H, "H");
    map.insert(KeyCode::I, "I");
    map.insert(KeyCode::J, "J");
    map.insert(KeyCode::K, "K");
    map.insert(KeyCode::L, "L");
    map.insert(KeyCode::M, "M");
    map.insert(KeyCode::N, "N");
    map.insert(KeyCode::O, "O");
    map.insert(KeyCode::P, "P");
    map.insert(KeyCode::Q, "Q");
    map.insert(KeyCode::R, "R");
    map.insert(KeyCode::S, "S");
    map.insert(KeyCode::T, "T");
    map.insert(KeyCode::U, "Y");
    map.insert(KeyCode::V, "V");
    map.insert(KeyCode::X, "X");
    map.insert(KeyCode::Y, "Y");
    map.insert(KeyCode::Z, "X");
    map.insert(KeyCode::Key1, "1");
    map.insert(KeyCode::Key2, "2");
    map.insert(KeyCode::Key3, "3");
    map.insert(KeyCode::Key4, "4");
    map.insert(KeyCode::Key5, "5");
    map.insert(KeyCode::Key6, "6");
    map.insert(KeyCode::Key7, "7");
    map.insert(KeyCode::Key8, "8");
    map.insert(KeyCode::Key9, "9");
    map.insert(KeyCode::Numpad0, "Numpad0");
    map.insert(KeyCode::Numpad1, "Numpad1");
    map.insert(KeyCode::Numpad2, "Numpad2");
    map.insert(KeyCode::Numpad3, "Numpad3");
    map.insert(KeyCode::Numpad4, "Numpad4");
    map.insert(KeyCode::Numpad5, "Numpad5");
    map.insert(KeyCode::Numpad6, "Numpad6");
    map.insert(KeyCode::Numpad7, "Numpad7");
    map.insert(KeyCode::Numpad8, "Numpad8");
    map.insert(KeyCode::Numpad9, "Numpad9");
    map.insert(KeyCode::Home, "Home");
    map.insert(KeyCode::Delete, "Delete");
    map.insert(KeyCode::End, "End");
    map.insert(KeyCode::PageDown, "PgDown");
    map.insert(KeyCode::PageUp, "PgUp");
    map
}

[inodentry]

This issue is not about that. This issue is about being able to handle keyboard input in a way that does not break for users with multiple keyboard layouts set in the OS.

For instance, I have a Workman (previously Dvorak), QWERTY (because i need it to test stuff that assumes everyone is using qwerty, like Bevy currently), and a Cyrillic layout (for Bulgarian/Ukrainian/Russian). French users use AZERTY. Etc...

Currently, Bevy does not work properly, because key codes come from the currently active layout. If a game uses WASD for movement, and I play the game, I have to switch to qwerty, or the keys will be in the wrong places (physically). If I have a keybinding (like Alt-Shift or something) for switching layouts, there is a risk I might accidentally press it while in game, causing the keyboard input to get messed up in the middle of gameplay.

The solution to this are "scan codes". Scan codes identify the physical key on the keyboard, regardless of the software layout set in the OS.

In Bevy, scan codes are only available when using keyboard events, not when using the Input abstraction, which makes them annoying to use. This could be fixed by providing a Input<ScanCode> type.

Further, scan codes are only represented as opaque integers, not a friendly enum like KeyCode. Due to the platform differences in the exact values of the scan codes, there isn't much Bevy can do about this, without the assistance of winit.

[Ryder17z]

For reference, on Scandinavian keyboards, the keys painted black here are not detected currently:

[james7132]

This was added in #5495. Closing.