Simplify non qwerty layout use

[crazyiop]

This PR gives a way to simply use another layout than qwerty in the layers definition.
There are more details in the doc change.

I have somehow tested it with my 1-key breadboard on multiple cases. (I have not yet received everything to build my keyboard...)

[kdb424]

I like the intent of this, but I think we may want to look at adding "system language" at a deeper level, rather than a translation layer that's special for "non US". Maybe we look at setting a language at boot time, or possibly even run time so you can just inform the firmware what language your PC is in, and it will have a table for that language, as opposed to assuming that QWERTY is the default? @xs5871 @klardotsh I'd like your opinion here.

[crazyiop]

I can see the benefit of switching dynamicaly the layout as I have seen a lot of keymap shenanigan in qmk to artificialy make a dvorak/colemak layout in addition to being able to use the standart ones (the mess is often with an artificial alt-gr layer or even shift layer to disociate (normal key, shifted key, altgr key) for symbols)

But I'm not sure it will be easy if event doable as the keyboard cannot know the os configuration.

as opposed to assuming that QWERTY is the default

To my understanding, every keyboard already does this assumption. Every keyboard send the same code for the same key. It is up to the os layout configuration to do the conversion. As the keyboard cannot know the os configuration it is not easily achievable.

The 'normal' way user switch layout is not by a switch on the keyboard, but by changing the layout at the os. Windows have a sys tray icon/list when having multiple layout configured (and a keyboard shortcut). On linux it is as simple as $ set xkbmap fr bepo so it can be automatize in different form...

As I said some people using qmk have some twisted layout to use a switch on the keyboard, but they in that case still assumed that their os is always in a specific layout (not always the qwerty one)...

[xs5871]

I'd like to link this issue to #430: make scancode lookups more flexible.

[klardotsh]

I'm not a good fit to review this, really: I not only get per-input-device key mapping on Linux+Wayland with libinput (which is static by device identifier), but also write in no other scripts that a standard QWERTY-bound layout can't handle (or at worst, if I'm trying to write German or French, I can get it done with a Compose key). I map, for example, laptop keyboards to Dvorak in my Sway config, but my KMK and QMK boards all fall back to the default QWERTY which I then make Dvorak in hardware. At UEFI and TTY, all keyboards are just QWERTY on the software side to keep my life sane (and avoid the problem laid out upthread).

[crazyiop]

@kdb424 I can wait for #430 and try to move my mapping as alternative of that KEY_GENERATION_FUNCTIONS but I have no big hope as it will hit some limitation:

• the keyboard still always communicate to the pc via the ascii standard (that is closely related to qwerty). Most of the KC.xx you need in your qwerty keymap layers are direct basic key in the ascii table. But other language with more char need to rely extensively on 'base' key being KC.RALT(KC.yy) or even KC.LSFT(KC.RALT(KC.zz)) who from what I got are not a handled case by the maybe_make_xx functions.
• for the multi layout capable keyboard: I cannot think of a proper way to "tell the keyboard" the os-layout that is not dedicated hardware. I'm certain only few people will go through the hassle to add such hardware and the mass will just keep having a sane setup: switching the layout at os level when necessary.

The os layout is here for a reason and does its job extremely well in some case. For example, with my current layout I have a dead µ (on alt-gr u) , so I can type µ+a = α in just a few stroke (works for the whole greek alphabet). Nothing about that is due to my current qmk firmware. Trying to cram all the layout handling inside kmk alone is not a good idea imho. It will be a big challenge to just get on par (eating all the micro memory in the process), and will probably just mess with the os layout processing anyway.

[kdb424]

@kdb424 I can wait for #430 and try to move my mapping as alternative of that KEY_GENERATION_FUNCTIONS but I have no big hope as it will hit some limitation:

* the keyboard still always communicate to the pc via the ascii standard (that is closely related to qwerty). Most of the `KC.xx` you need in your qwerty keymap layers are direct basic key in the ascii table. But other language with more char need to rely extensively on 'base' key being `KC.RALT(KC.yy)` or even `KC.LSFT(KC.RALT(KC.zz))` who from what I got are not a handled case by the `maybe_make_xx` functions.

* for the multi layout capable keyboard: I cannot think of a proper way to "tell the keyboard" the os-layout that is not dedicated hardware. I'm certain only few people will go through the hassle to add such hardware and the mass will just keep having a sane setup: switching the layout at os level when necessary.

The os layout is here for a reason and does its job extremely well in some case. For example, with my current layout I have a dead µ (on alt-gr u) , so I can type µ+a = α in just a few stroke (works for the whole greek alphabet). Nothing about that is due to my current qmk firmware. Trying to cram all the layout handling inside kmk alone is not a good idea imho. It will be a big challenge to just get on par (eating all the micro memory in the process), and will probably just mess with the os layout processing anyway.

That seems reasonable. I was rusty on how the OS handled that, and did some digging and you seem to be correct. I only use an IME for 日本語 and didn't remember easily how other languages worked, but yeah, that makes sense.

[crazyiop]

I have finished my handwired board and as I now really use the firmware I made some minor change in the new name for the key. also include small change in the generating script.

I am still convinced that basing new layout on the existing KC is the right choice. I did not really investigate replacing directly it... This works great as is. (first complete message typed on this new kmk powered keyboard) \o/

[kdb424]

I'm not sure if I'm qualified to respond to this in a constructive way. Hoping that either @klardotsh or @xs5871 have some input here.

[xs5871]

What is the benefit over a way simpler key-override like it's done in 'extensions/keymap_extras/keymap_jp.py'?

[crazyiop]

@xs5871 It is doable, but seems more complicated to me for a complete language layout. The jp keymap is already not that easy to follow imho as some key are reused multiple time, it also don't include a full circle permutation.

Some things to consider I can think of here:

• the key referred (typically base letter) will get updated at some point in the file, and reuse later making the order of those definition have a huge impact of the result (in the jp keymap, they are already a lot of constraint "this line before this other"). -> might be completely handled by the generating script from qmk files

BP.U = KC.S
BP.UGRAVE = BP.RALT(KC.S)
# becomes:
KC.U = KC.S
KC.UGRAVE = KC.RALT(KC.U)

• moving the letters key around will end up in a number of permutations, so we well have to use temp variable or swap like KC.A, KC.Q = KC.Q, KC.A of different length. -> totally acceptable, but not great to read when it becomes a 5/6 lenght circle.
• The ability to have layers using different language: being close to the german border I know peoples who cross the border every day to go to work. -> very narrow if even really existing use case

None of them are real blocker, my main point is that not overriding anything makes it less error-prone when making/checking those layout that are already annoying enough. I think the current suggestion is more explicit and therefor easier to follow/work with.

[xs5871]

Sure, let's get it in as an intermediate solution and maybe revisit once #430 is merged.

But: the script is something I would consider a 3d-party tool, in the sense that it's not relevant to the firmware per se.
Publish it by yourself and we'd be happy to link to it in the documentation.

[crazyiop]

I have moved the script in its own repo as asked. Linked in the doc.

[xs5871]

There are unaddressed change requests and questionable changes.