Add support for intl key legends based on the host keyboard layout #1161

precondition · 2022-08-08T17:03:40Z

Description

demo-intl-configurator.webm

(I capped the framerate at 20fps to reduce the file size, that's why it looks choppy)

I replaced the ISO toggle switch in the settings by a more general dropdown that lets the users pick the correct logical/functional keyboard layout of their OS. The only effect of this setting is to change how each keycode is displayed; nothing about the constructed keymap.json is altered.

To get started, I added the following host keyboard layouts:

US
UK
German
Russian

The lookup tables used for those host keyboard layouts can be found in src/i18n/keymap_extras and were generated with the help of the src/i18n/keymap_extras/convert_keymap_extras_header.js script. The script would allow me to easily add support for every keymap_extra input language that qmk_firmware supports, but for the sake of keeping this PR more concise, I limited myself to four OS keyboard layouts.

The bulk of the relatively impressive amount of added lines in this PR comes from those generated lookup tables, and to a lesser extend from the optional converter script used to generate them. The actual core changes to qmk_configurator are small.

Future Work

Can be addressed in follow-up PRs

Support more OS layouts (See Add more OS keyboard layouts #1165)
Add the corresponding "host_language" in the keymap.json
Guess the user's OS keyboard layout based on their locale
Possibly hide keys for capital letters in the "Shifted Symbols" section
Include comments and ASCII art in the generated output of convert_keymap_extras.py
Document the process of adding support for a new keymap_extras/OS keyboard layout
Sort the OS kb layout drop-down menu by localised/translated labels (Done in cc349e1)
Remove all dead code related to the defunct ISO toggle
Support localisation for the magic swap keycodes in the convert_keymap_extras_header.js script

Known Bugs

The text size for legends is computed based on the length of the key.name of the OS keyboard layout initially loaded and does not get updated when switching OS layouts until the user initiates a refresh. Concretely, this means that if the last saved OS layout was keymap_us, the name property of KC_SCLN would have been ":\n;". If the user switches to keymap_german, the locale keysym of KC_SCLN ("ö") will be shown in small text despite being a single character. ~~(Solved in b84a675)~~ Indirectly solved by Explore how to support internationalization of the keymap #1166
The test page (config.qmk.fm/#/test) doesn't dynamically update the key legends when switching OS layouts in the menu. You need to switch and reload the page for the new OS keyboard layout to apply. ~~(Solved in 1bad93e)~~ Indirectly solved by Explore how to support internationalization of the keymap #1166
Host Keyboard Layout dropdown is empty if an old configuratorSettings object missing the new osKeyboardLayout property is in localstorage. (Solved in 82cb033)

Motivation

Linked Issues

Feature Request: Add European layouts to the onscreen keyboard #866

[Feature Request] Support for other software keyboard layouts #846

Unable to add ISO-specific keys in configurator #768

Feature: Non--US Keyboard support #229

src/store/modules/keycodes/index.js

yanfali · 2022-08-10T01:14:13Z

I'm still digesting the PR, so I'm not ignoring you I promise. I think it looks pretty good, I just want to think about it.

I'm not thrilled to add python to the repo as it's even less portable than node in some respects.

noroadsleft

Likewise, I'm not ignoring you or this PR either; unfortunately you've caught me just a few days ahead of being out of town for six days. I probably won't be able to properly review this until next week.

yanfali · 2022-08-10T01:18:50Z

Sometimes, switching from an OS layout where a key has a shift pair to an OS layout where that same key is just a letter displays the letter in small script because it still thinks that the length of key.name is greater than 1. A refresh fixes it.

This may be related to how the key name is being updated. The UI won't render if the reference doesn't change, depending on how it's rendering. I haven't looked at this code in a long time, It might be as simple as re-assigning the object to a new one and copying all the fields. The styling code should react to length changes, or it could just be a bug.

precondition · 2022-08-10T04:21:18Z

I'm not thrilled to add python to the repo as it's even less portable than node in some respects.

I can translate the script into node.js if needs be. It is mostly text and hashmap manipulation so it is not hard to port.

src/i18n/keymap_extras/index.js

precondition · 2022-08-11T21:31:42Z

If we ignore the deprecated keymap_nordic.h and the steno/plover header files that are out-of-scope, the converter script generates (syntaxically) correct output for every^† keymap_*.h header in qmk_firmware/quantum/keymap_extras!

^†: Assuming that one merges the PRs I opened to fix the bugs in keymap_bepo.h (qmk/qmk_firmware#17999) and keymap_lithuanian_qwerty.h (qmk/qmk_firmware#18000)
PS: Turns out there are more.

yanfali · 2022-08-12T00:28:12Z

If we ignore the deprecated keymap_nordic.h and the steno/plover header files that are out-of-scope, the converter script generates (syntaxically) correct output for every† keymap_*.h header in qmk_firmware/quantum/keymap_extras!

†: Assuming that one merges the PRs I opened to fix the bugs in keymap_bepo.h (qmk/qmk_firmware#17999) and keymap_lithuanian_qwerty.h (qmk/qmk_firmware#18000)

Thanks for doing this. I am traveling at the moment, but I will take a look this weekend. I need to test it myself.

src/i18n/keymap_extras/convert_keymap_extras_header.js

precondition · 2022-08-12T10:29:29Z

While working on localising QK_GESC, I realised that its wording can be misleading. "Esc normally, but ~ when Shift or GUI is pressed". First of all, the Shift or GUI modifiers don't necessarily need to be pressed while pressing QK_GESC, it would work just fine even if you tapped a one-shot modifier. The important point is that one of those modifiers needs to be active. Secondly, when GUI is active, QK_GESC produces ` (at least on US), not ~.

My suggestion:

diff --git a/src/store/modules/keycodes/quantum.js b/src/store/modules/keycodes/quantum.js
index 734e9a1..90c5921 100644
--- a/src/store/modules/keycodes/quantum.js
+++ b/src/store/modules/keycodes/quantum.js
@@ -392,9 +392,9 @@ export default [
   { label: 'Special action keys', width: 'label' },
 
   {
-    name: 'Esc / ~',
+    name: '` / ~\nEsc',
     code: 'QK_GESC',
-    title: 'Esc normally, but ~ when Shift or GUI is pressed'
+    title: 'Esc normally, but ` when GUI is active or ~ when Shift is active'
   },
   {
     name: 'LS / (',

Displaying the legend on 2 lines would also be more coherent with the rest of the shift pairs. If you think about it, the shift pair of QK_GESC is base=Esc and Shift=~ after all. Although I'm not entirely convinced by the ` / part.

See commit 5b36bf9

This is the main commit of PR qmk#1161.

yanfali · 2022-08-12T23:12:56Z

While working on localising QK_GESC, I realised that its wording can be misleading. "Esc normally, but ~ when Shift or GUI is pressed". First of all, the Shift or GUI modifiers don't necessarily need to be pressed while pressing QK_GESC, it would work just fine even if you tapped a one-shot modifier. The important point is that one of those modifiers needs to be active. Secondly, when GUI is active, QK_GESC produces ` (at least on US), not ~.

My suggestion:
diff --git a/src/store/modules/keycodes/quantum.js b/src/store/modules/keycodes/quantum.js
index 734e9a1..90c5921 100644
--- a/src/store/modules/keycodes/quantum.js
+++ b/src/store/modules/keycodes/quantum.js
@@ -392,9 +392,9 @@ export default [
   { label: 'Special action keys', width: 'label' },
 
   {
-    name: 'Esc / ~',
+    name: '` / ~\nEsc',
     code: 'QK_GESC',
-    title: 'Esc normally, but ~ when Shift or GUI is pressed'
+    title: 'Esc normally, but ` when GUI is active or ~ when Shift is active'
   },
   {
     name: 'LS / (',
 
Displaying the legend on 2 lines would also be more coherent with the rest of the shift pairs. If you think about it, the shift pair of QK_GESC is base=Esc and Shift=~ after all. Although I'm not entirely convinced by the / ~ part.

See commit 5b36bf9

I have no opinion on this, as it doesn't work on macOS at all due to a different binding policy. @qmk/collaborators may have other opinions.

precondition · 2022-08-13T18:29:28Z

I just realised that since I mostly rely on the old pre-existing ISO toggle mechanism to update the key legends text, the test page (config.qmk.fm/#/test) doesn't dynamically update the key legends when switching OS layouts in the menu (since the old ISO toggle has no effect on that page). You need to switch and reload the page for the new OS keyboard layout to apply.

I have a feeling that fixing the first known bug will also patch this. However, I'm in the middle of a move right now and I will not have Wi-Fi in the next few days so I don't know when I'll be able to look into the issue.

yanfali · 2022-08-16T22:40:06Z

I just realised that since I mostly rely on the old pre-existing ISO toggle mechanism to update the key legends text, the test page (config.qmk.fm/#/test) doesn't dynamically update the key legends when switching OS layouts in the menu (since the old ISO toggle has no effect on that page). You need to switch and reload the page for the new OS keyboard layout to apply.

I have a feeling that fixing the first known bug will also patch this. However, I'm in the middle of a move right now and I will not have Wi-Fi in the next few days so I don't know when I'll be able to look into the issue.

This never needed to work iirc as you could

I just realised that since I mostly rely on the old pre-existing ISO toggle mechanism to update the key legends text, the test page (config.qmk.fm/#/test) doesn't dynamically update the key legends when switching OS layouts in the menu (since the old ISO toggle has no effect on that page). You need to switch and reload the page for the new OS keyboard layout to apply.

I have a feeling that fixing the first known bug will also patch this. However, I'm in the middle of a move right now and I will not have Wi-Fi in the next few days so I don't know when I'll be able to look into the issue.

As there are explicit ISO and ANSI buttons it was never really a priority to make this toggle either. It wouldn't be too hard to fix this though.

yanfali · 2022-08-16T23:38:47Z

super busy at work. I'll try and make time for this on thursday

yanfali · 2022-08-18T19:38:34Z

Thank you for all the hardwork. I will be reviewing this properly today, evening PST.

precondition · 2022-08-19T16:08:50Z

As I've guessed, the two known bugs share a common root cause. The root cause is that the values of this.meta (and by extension, this.meta.name) are set according to the OS keyboard layout selected during the initial loading of the web page, and does not get updated when the user switches OS layouts in the settings panel. Consequently, any components like BaseKey.vue or TesterKey.vue that rely on the value of this.meta.name to carry out their tasks do not react appropriately to the new OS layout settings (until a refresh of the page is made).

My solution, as presented in commits b84a675 and 1bad93e, is to use const { name } = this.lookupKeycode(this.meta.code); instead of this.meta.name. It works but I'm not a big fan of paying the O(n) price of the lookupKeycode call (due to state.keycodes.find()). Ideally, this.meta.name should get updated when the user switches OS keyboard layouts so that we can stay in O(1).

yanfali · 2022-08-19T19:45:00Z

There's an issue for the very first time it loads, the localstorage stuff is out of sync until the first reload.

src/store/modules/app/mutations.js

yanfali · 2022-08-19T19:54:07Z

when switching to russian the ISO big enter doesn't kick in. Is that the expected behavior?

precondition · 2022-08-19T19:59:25Z

when switching to russian the ISO big enter doesn't kick in. Is that the expected behavior?

Yes, the Russian keymap is set to be ANSI because the OS layout produces the same keysym for KC_BSLS, KC_NUHS and KC_NUBS. Additionally, the keymap_russian.h file included in qmk_firmware clearly shows an ANSI ASCII art:

/*
 * ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───────┐
 * │ Ё │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │ 0 │ - │ = │       │
 * ├───┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─────┤
 * │     │ Й │ Ц │ У │ К │ Е │ Н │ Г │ Ш │ Щ │ З │ Х │ Ъ │  \  │
 * ├─────┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴─────┤
 * │      │ Ф │ Ы │ В │ А │ П │ Р │ О │ Л │ Д │ Ж │ Э │        │
 * ├──────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴────────┤
 * │        │ Я │ Ч │ С │ М │ И │ Т │ Ь │ Б │ Ю │ . │          │
 * ├────┬───┴┬──┴─┬─┴───┴───┴───┴───┴───┴──┬┴───┼───┴┬────┬────┤
 * │    │    │    │                        │    │    │    │    │
 * └────┴────┴────┴────────────────────────┴────┴────┴────┴────┘
 */

Thanks Markus

src/store/modules/keycodes/index.js

src/components/SettingsPanel.vue

yanfali

I think this is looking really good. I'll try and re-test it this weekend and I think we can probably get it merged then. Thank you for your hard work and patience.

…deNames`

noroadsleft · 2022-08-25T23:34:35Z

Can you try again with commit ac76998 ? After this commit, I can't reproduce the issue you describe but it wouldn't be the first time that the drop-down suffers from unforeseen bugs.

The changes should in theory prevent other similar new-property-missing-from-locally-stored-settings problems in the future.

Issue is fixed. ✔️

src/components/SettingsPanel.vue

yanfali · 2022-08-28T16:57:54Z

src/store/modules/app/state.js

+    // so it is important to merge them with the default configurator settings to ensure that all
+    // settings are set.
+    let conf = {
+      ...getDefaultConfiguratorSettings(),


src/store/modules/keycodes/index.js

yanfali

A few minor changes need to manipulating the configurator saved state, otherwise LGTM.

yanfali · 2022-08-28T17:05:10Z

@precondition I think we're almost ready to merge this. Thanks again for this. Very cool feature. Just a couple of very small clean ups and we're good to go.

This is done so that the Shift+GESC and GUI+GESC keysyms/keycode names remain on the same line because the `breakLines` functions of PrintKey.vue and TesterKey.vue replace spaces ' ' with newlines '\n'.

yanfali

Lgtm. @noroadsleft can you give it one more test. Otherwise I think it's ready to merge.

yanfali · 2022-09-03T22:40:01Z

@noroadsleft bump

noroadsleft

Looks good to me.

Thanks for this; been wanting this feature for a long time.

precondition · 2022-09-04T11:02:59Z

Should the linked issues be closed now?

yanfali reviewed Aug 10, 2022

View reviewed changes

src/store/modules/keycodes/index.js Outdated Show resolved Hide resolved

noroadsleft reviewed Aug 10, 2022

View reviewed changes

precondition force-pushed the intl_support branch from 7b3f406 to bbd6dc9 Compare August 10, 2022 10:37

precondition commented Aug 10, 2022

View reviewed changes

src/i18n/keymap_extras/index.js Outdated Show resolved Hide resolved

yanfali reviewed Aug 12, 2022

View reviewed changes

src/i18n/keymap_extras/convert_keymap_extras_header.js Outdated Show resolved Hide resolved

precondition force-pushed the intl_support branch from 7f97d24 to fa18fe3 Compare August 12, 2022 10:13

precondition force-pushed the intl_support branch from fa18fe3 to eed17d7 Compare August 12, 2022 11:46

precondition added a commit to precondition/qmk_configurator that referenced this pull request Aug 12, 2022

Add support for intl key legends based on the host keyboard layout

1d90e84

This is the main commit of PR qmk#1161.

precondition force-pushed the intl_support branch from e3b5406 to c8a9da5 Compare August 12, 2022 22:03

precondition added a commit to precondition/qmk_configurator that referenced this pull request Aug 12, 2022

Add support for intl key legends based on the host keyboard layout

86aa319

This is the main commit of PR qmk#1161.

precondition force-pushed the intl_support branch from c8a9da5 to 15bd89c Compare August 12, 2022 22:10

precondition force-pushed the intl_support branch from bdd4a65 to 7a22895 Compare August 13, 2022 16:29

precondition mentioned this pull request Aug 14, 2022

Remove duplicate COMBINING HORN in keymap_us_extended.h qmk/qmk_firmware#18045

Merged

14 tasks

precondition mentioned this pull request Aug 19, 2022

Add more OS keyboard layouts #1165

Merged

yanfali reviewed Aug 19, 2022

View reviewed changes

src/store/modules/app/mutations.js Show resolved Hide resolved

precondition added 3 commits August 23, 2022 21:21

Sort osKeyboardLayout translatable strings more consistently

f82764c

Translate options to German

7a4c67a

Thanks Markus

Use a more formal verb (coller→correspondre)

b9ac8b2

yanfali reviewed Aug 25, 2022

View reviewed changes

src/store/modules/keycodes/index.js Show resolved Hide resolved

yanfali reviewed Aug 25, 2022

View reviewed changes

src/store/modules/keycodes/index.js Outdated Show resolved Hide resolved

yanfali reviewed Aug 25, 2022

View reviewed changes

src/components/SettingsPanel.vue Outdated Show resolved Hide resolved

yanfali reviewed Aug 25, 2022

View reviewed changes

precondition added 2 commits August 25, 2022 16:55

Change active keycode picker tab via changeActive, not `updateKeyco…

eaee8fa

…deNames`

Catch promise exceptions when setting lang or OS kb layout

414986d

yanfali reviewed Aug 28, 2022

View reviewed changes

src/components/SettingsPanel.vue Outdated Show resolved Hide resolved

yanfali reviewed Aug 28, 2022

View reviewed changes

src/store/modules/keycodes/index.js Outdated Show resolved Hide resolved

yanfali reviewed Aug 28, 2022

View reviewed changes

src/store/modules/keycodes/index.js Outdated Show resolved Hide resolved

yanfali requested changes Aug 28, 2022

View reviewed changes

precondition added 4 commits August 28, 2022 19:39

Use string template for osLayout label

fee4d15

*Commit* changes to ISO in generateKeycodes

2f5c48a

Use commit and getters to manipulate OS layout in getOSKeyboardLayout

8cb10a2

Remove spaces around the '/' of KC_GESC's keycode name

bc7b3b3

This is done so that the Shift+GESC and GUI+GESC keysyms/keycode names remain on the same line because the `breakLines` functions of PrintKey.vue and TesterKey.vue replace spaces ' ' with newlines '\n'.

precondition requested a review from yanfali August 28, 2022 18:51

yanfali approved these changes Aug 28, 2022

View reviewed changes

Use string template for translatedLayoutName

b31875b

noroadsleft approved these changes Sep 4, 2022

View reviewed changes

noroadsleft added enhancement feature labels Sep 4, 2022

noroadsleft merged commit 165258a into qmk:master Sep 4, 2022

precondition deleted the intl_support branch September 4, 2022 10:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for intl key legends based on the host keyboard layout #1161

Add support for intl key legends based on the host keyboard layout #1161

precondition commented Aug 8, 2022 •

edited

Loading

yanfali commented Aug 10, 2022

noroadsleft left a comment

yanfali commented Aug 10, 2022 •

edited

Loading

precondition commented Aug 10, 2022

precondition commented Aug 11, 2022 •

edited

Loading

yanfali commented Aug 12, 2022

precondition commented Aug 12, 2022 •

edited

Loading

yanfali commented Aug 12, 2022

precondition commented Aug 13, 2022 •

edited

Loading

yanfali commented Aug 16, 2022

yanfali commented Aug 16, 2022

yanfali commented Aug 18, 2022

precondition commented Aug 19, 2022 •

edited

Loading

yanfali commented Aug 19, 2022

yanfali commented Aug 19, 2022

precondition commented Aug 19, 2022

yanfali left a comment

noroadsleft commented Aug 25, 2022

yanfali Aug 28, 2022

yanfali left a comment

yanfali commented Aug 28, 2022

yanfali left a comment

yanfali commented Sep 3, 2022

noroadsleft left a comment

precondition commented Sep 4, 2022

Add support for intl key legends based on the host keyboard layout #1161

Add support for intl key legends based on the host keyboard layout #1161

Conversation

precondition commented Aug 8, 2022 • edited Loading

Description

Future Work

Known Bugs

Motivation

Linked Issues

yanfali commented Aug 10, 2022

noroadsleft left a comment

Choose a reason for hiding this comment

yanfali commented Aug 10, 2022 • edited Loading

precondition commented Aug 10, 2022

precondition commented Aug 11, 2022 • edited Loading

yanfali commented Aug 12, 2022

precondition commented Aug 12, 2022 • edited Loading

yanfali commented Aug 12, 2022

precondition commented Aug 13, 2022 • edited Loading

yanfali commented Aug 16, 2022

yanfali commented Aug 16, 2022

yanfali commented Aug 18, 2022

precondition commented Aug 19, 2022 • edited Loading

yanfali commented Aug 19, 2022

yanfali commented Aug 19, 2022

precondition commented Aug 19, 2022

yanfali left a comment

Choose a reason for hiding this comment

noroadsleft commented Aug 25, 2022

yanfali Aug 28, 2022

Choose a reason for hiding this comment

yanfali left a comment

Choose a reason for hiding this comment

yanfali commented Aug 28, 2022

yanfali left a comment

Choose a reason for hiding this comment

yanfali commented Sep 3, 2022

noroadsleft left a comment

Choose a reason for hiding this comment

precondition commented Sep 4, 2022

precondition commented Aug 8, 2022 •

edited

Loading

yanfali commented Aug 10, 2022 •

edited

Loading

precondition commented Aug 11, 2022 •

edited

Loading

precondition commented Aug 12, 2022 •

edited

Loading

precondition commented Aug 13, 2022 •

edited

Loading

precondition commented Aug 19, 2022 •

edited

Loading