Modifiers in modifier functions get dropped in certain programs/desktop environments

[caksoylar]

Edited to make it more generic: This issue is concerning the timing of keys sent by ZMK when modifier functions are involved, such as &kp LA(TAB), &kp HASH (equal to &kp LS(N3)) etc. Some applications/desktop environments don't seem to like how ZMK sends these keys and sometimes ignores the modifier parts of these keycodes, leading to unexpected behavior. So far issues have been observed in:

• Windows classic RDP app (see original post below)
• Gnome under Wayland (upstream issue linked below)

Original post

There are issues using sending key presses with modifier functions and overall issues with held modifiers when using the RDP app ("Remote Desktop Connection") on Windows 10. I use it for connecting to a local VM with Hyper-V using vmconnect.exe, so there is no network latency. For instance when sending a key press like LALT(TAB) the modifier is almost always ignored. When using regular modifiers like Shift, they are also not detected some of the time.

Note that there is also a different RDP app ("Remote Desktop Classic" on MS Store) that doesn't seem to have the issue, so it is likely related to the peculiar handling of keyboard inputs of the former app. However this app doesn't allow connections to Hyper-V VMs.

Here is me sending &kp UNDER in the host (not using RDP) on https://config.qmk.fm/#/test:

Here it is on the remote VM through the classic RDP app:

Worth noting that the minus keycode is also reported differently, but I don't know if that has any relevance.

Could it potentially be helped by a parameter like tap-duration-ms as proposed in #756? Or maybe the RDP app expects keycodes to be spaced apart in time?

[okke-formsma]

Could you show the timings when you press a keycode like that in QMK? The solution for #756 does not apply here, as that is for hold-taps and does not apply to shifted keycodes.

[caksoylar]

This is a bit interesting, here is QMK underscore keycode on local (no RDP), one short tap and one long tap:

This is the same on remote:

First of all the order of key ups are different, QMK seems to release shift ~16ms after the minus, but ZMK releases shift before the minus (they seem to be held for the exact same duration).

I also have issues getting manual shift holds recognized with ZMK if I am not pressing it very deliberately which doesn't happen with QMK. Let me know if there is anything I can share to help debug that.

[okke-formsma]

Are you using ZMK while connected through bluetooth or usb? Could you verify the issue is the same using both connection methods?

It seems QMK is not processing the shift and '-' keyup events quickly, but adding some 16ms delays in between. To emulate this we may have to somehow queue keypresses instead of processing them as fast as possible.

I've never had issues with 'deliberate pressing shift' or not, so can't really say anything about that.

[caksoylar]

Are you using ZMK while connected through bluetooth or usb? Could you verify the issue is the same using both connection methods?

I primarily use BT but I can confirm that the behavior is the same on USB.

I've never had issues with 'deliberate pressing shift' or not, so can't really say anything about that.

To clarify this is only over RDP; I don't have any such issues outside the RDP app.

I tried the short tap and long tap for &kp UNDER again over USB, on the remote VM. The key-up event for Shift ~5ms after key-down is very consistent:

I also noticed that if I hold the &kp UNDER button down long enough (>500ms), it detects an additional shift down event as in the long tap event above (red highlighted part). This doesn't happen outside RDP or with QMK inside or outside RDP.

I guess the RDP client is doing its own held key detection with some thresholds and looking at event ordering, and ZMK's order of events somehow doesn't play well with it.

[halfurness]

I'm seeing this behaviour too, LC() isn't working for me in RDP

[caksoylar]

Potentially related: qmk/qmk_firmware#13708

[abstracterror]

A workaround that mostly seems to work for individual &kp bindings is to define a macro for each that presses the modifier key, taps the unmodified key code, and then releases the modifier key.

I set wait-ms to 16ms in the macros, based on the numbers given in the comments above. I still get occasional unshifted characters, but much less frequently.

Unfortunately the issue also applies to &caps_word, with some of the letters ending up shifted, and some not.

[rambip]

I have a similar timing issue on gnome (wayland + fedora)
I defined a "tap-modifier" to output colon, but in gnome it outputs semicolon

But even stranger, in sway (a wayland tiling window manager) I never get tbat error.

I get it with bluetooth and usb.

Are there values I can tweak in the config for timing ?

[catdevnull]

@rambip I have a similar problem with sticky keys; in Sway they are reliable, but in GNOME they are hit or miss.

[catdevnull]

Sorry for the shitty video, but here's a demo:

muestra2.webm

[catdevnull]

Even in GNOME, it doesn't happen under Qt apps. This also affects things like &kp AT, yielding 2 instead.

[hallettj]

Because I'm having the same problem in Gnome & Wayland I used @abstracterror's helpful macro workaround. I agree it makes modified key presses work much more consistently, but it is still not perfect. I didn't want to write a macro for every shifted key I use, so I wrote this C preprocessor macro that generates ZMK macros:

#define SHIFTED(sym, macro_label) s_##sym: s_##sym {\
        label = macro_label;\
        compatible = "zmk,behavior-macro";\
        wait-ms = <50>;\
        #binding-cells = <0>;\
        bindings\
            = <&macro_press &kp LSHFT>\
            , <&macro_press &kp sym>\
            , <&macro_release &kp LSHFT>\
            , <&macro_pause_for_release>\
            , <&macro_release &kp sym>\
            ;\
    }

My macros section looks like this:

    macros {
        SHIFTED(EXCL, "shifted_excl");
        SHIFTED(AT, "shifted_at");
        SHIFTED(HASH, "shifted_hash");
        /* etc. */
    };

That lets me use bindings of the form &s_EXCL, &s_AT, &s_HASH, etc.

I'm passing in the shifted codes (like EXCL instead of N1) which is technically redundant. That helps me to keep my configuration readable, and it doesn't seem to hurt anything.

I tried to derive the label from the first macro argument, but I couldn't figure out how to get macro-level string interpolation or concatenation to work. So I gave up, and used a second argument for the label.

I hope this is helpful for someone in a similar situation.

Edit 2022-12-15: I edited the preprocessor macro above to release LSHFT before the pause_for_release step to prevent the shift modifier from bleeding into follow-up key presses. I also increased the wait time from 35 to 50 ms because I'm still seeing occasional unshifted characters.

[catdevnull]

As far as I can tell, this is an upstream issue: https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/5890

[pergamomo]

Until this is fixed in Gnome is there a way to adjust the internal timing globally so we don't need to write a macro for every shifted key?

[catdevnull]

Annoyingly, it doesn't seem to affect QMK :(

[hallettj]

As far as I can tell, this is an upstream issue: https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/5890

I've just noticed a comment in that issue that suggests running pkill ibus as a workaround to the problem in Gnome. That seems to work for me!

It looks like ibus provides features like input methods for alphabets other than the Latin one, and for emoji. I'm not sure, but it might also be required for switching input methods. I don't know much about the consequences of disabling it. I also don't see any built-in option to disable it permanently. Comments I'm seeing in searches say that "Gnome requires ibus".

[pergamomo]

@hallettj Sadly this didn't work for me. Maybe because I'm using English (intl. with AltGr dead keys)?

Killing iBus also killed the keyboard input so I had to reboot.

[catdevnull]

@pergamomo I'm the original poster of that workaround, and I'm using the same layout. Not sure what ibus really does, but I read something about input methods and thought that this could be the problem.

[JMHAVANCE]

Not sure if this helps anyone, but using Remote Desktop on Windows (in windowed mode) solved this issue for me connecting to both other Windows machines as well as Ubuntu machines.

Full screen: issues with hold-tap and layers.
Windowed: normal

(even when stretching the window across my entire screen)

[caksoylar]

Also relevant: qmk/qmk_firmware#19449

From the PR description it sounds like sending modifier events separate from other key events helps with both RDP and Gnome scenarios.

[abliss]

@JMHAVANCE I have a theory about why it works for you in windowed mode but not fullscreen: There's an option in the windows RDP settings called something like "Apply windows hotkeys locally (e.g. alt-tab)". (In the config file, I think this is "keyboardhook"). It has three values: "always" (1), "never" (0), and "only when full-screen" (??). Maybe you have "only when full-screen" selected and thus windowed/fullscreen makes the difference for you? For me, I found that setting this value to "never" fixes the problem even when I am fullscreen (which I basically have to be, because the weird layout of my monitors makes it impossible for a window to cover all/only my active pixels.)

[manna-harbour]

Is there a way to modify the timing as QMK does with TAP_CODE_DELAY?

#1443

#1444 (comment)

[kevinpastor]

@manna-harbour, I've looked what MIRYOKU_KLUDGE_TAPDELAY enables in Miryoku but can't grasp what it actually does. From my understanding, you're defining a macro that simply presses and releases a given key. Are you overwritting the default constants ZMK is providing for letters and some symbols (e.g. A or SQT)? I can't find where the wait time is defined in your repo; does this flag modifies the default wait time of a macro?

Any information would greatly help. Thanks in advance!

[kevinpastor]

I'm sorry, I didn't realize that the link I sent was about Mod-Tap specifically. Would there be something similar for modifier functions?

[kevinpastor]

I've tried a lot of things and finally realized that the workaround of a macro slowing things down did work over USB but not Bluetooth.

I am now wondering if one of the system configuration for Bluetooth could help here. I've tried to understand what CONFIG_ZMK_BLE_CONSUMER_REPORT_QUEUE_SIZE, CONFIG_ZMK_BLE_KEYBOARD_REPORT_QUEUE_SIZE, CONFIG_ZMK_BLE_THREAD_PRIORITY, and CONFIG_ZMK_BLE_THREAD_STACK_SIZE do by reading ZMK and Zephyr's documentation, but I'm totally lost. Could these settings help in the timing of keypresses? Setting CONFIG_ZMK_BLE_CONSUMER_REPORT_QUEUE_SIZE to 1 seemed to have helped a bit, but I'm not sure if it's unrelated.

For anyone wondering what my macro looks like, here's the snippet I use which makes use of zmk-nodefree-config helpers:

ZMK_BEHAVIOR(ls, macro_one_param, \
    wait-ms = <0>; \
    tap-ms = <0>; \
    bindings = \
        <&macro_press &kp LSHFT>, \
        <&macro_wait_time 100>, \
        <&macro_param_1to1>, \
        <&macro_tap &kp MACRO_PLACEHOLDER>, \
        <&macro_wait_time 100>, \
        <&macro_release &kp LSHFT>; \
)

// Using the binding in the keymap
&ls N7

[caksoylar]

Note that &macro_wait_time 100 doesn't mean "wait here for 100ms", it means "change the wait time between actions to 100ms after this". So I am guessing the timing between shift press and param tap isn't working like you'd want it to. Looks like you can simply use wait-ms = <100>; and get rid of the macro_wait_time bindings. I'd also set tap-ms to a larger value like 10 if it still isn't working.

[kevinpastor]

Would setting wait-ms = <100>; cause a wait time after <&macro_param_1to1>? My initial reasoning was that I did not want a total wait time of ~300ms because of the three "sandwiched" bindings.

[caksoylar]

Would setting wait-ms = <100>; cause a wait time after <&macro_param_1to1>? My initial reasoning was that I did not want a total wait time of ~300ms because of the three "sandwiched" bindings.

I don't think it would since they are control bindings, not actual behaviors.

[teresi]

I found success with the following from abstracterror, kevinpastor, and caksoylar above

ZMK_BEHAVIOR(ls, macro_one_param, \
    wait-ms = <16>; \
    tap-ms = <0>; \
    bindings = \
        <&macro_press &kp LSHFT>, \
        <&macro_param_1to1>, \
        <&macro_tap &kp MACRO_PLACEHOLDER>, \
        <&macro_release &kp LSHFT>; \
)

&ls N8  // asterisk

I observed symbols getting converted to the un-shifted state in Alacritty,
when I hold a *lt <layer> and quickly tap a &kp <symbol> for one of these:

TILDE, EXCL, AT, HASH, DOLLAR, PERCENT, CARET, AMPERSAND, ASTERISK, LEFT_PARENTHESIS, RIGHT_PARENTHESIS, PIPE

Although if I press the layer-tap, wait a little, then press the &kp <symbol> it gives me the symbol correctly.

I'm using bluetooth, X11, gnome 42, and not using remote desktop.
If I use gnome-terminal or gedit the symbols output correctly.
If I set wait-ms=0 then the symbols get un-shifted almost every time.

[TristanCacqueray]

I've just noticed a comment in that issue that suggests running pkill ibus as a workaround to the problem in Gnome. That seems to work for me!

Thanks, that also works for me with gnome 44 and wayland.

[kevinpastor]

I just retried the vanilla setup with CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES and I'm still getting the issue over Bluetooth. Is there anything that could be done in ZMK instead of the macro hack on the user side?

The usage of the macro does not allow me to use mod-morph when the modifier key uses that macro. For example, I want a key where ( is sent by default and where ) is sent when Shift is held. Having the bindings to <&ls N9>, <&ls N0> only outputs 0 when Shift is held.

[caksoylar]

We are veering off topic here, but you don't need the macro for the second mod morph? You are already holding down shift, so define keep-mods = (MOD_LSFT|MOD_RSFT); in your mod morph and use bindings = <&ls N9>, <&kp N0>;.

[centy]

For windows remote desktop set:

1. Tab "Local Resources"
2. Section keyboard
3. pick "On the remote computer"

It works fine on my setup

[leonasdev]

This issue also happened while using Parsec immersive mode (like the fullscreen of rdp).

[Timoyoungster]

For windows remote desktop set:

1. Tab "Local Resources"
2. Section keyboard
3. pick "On the remote computer"

It works fine on my setup

in my case it works when picking "On this computer", not the remote one
(Windows RDP in both fullscreen and windowed mode)

[leonasdev]

In my case of Windows RDP, set to "On the remote computer" or "Only when using the fullscreen" not work, set to "On this computer" just make the alt+tab apply to the client computer not remote computer. But this issue is about alt+tab not working properly in "remote computer"...... so guys please stop saying set it to "On this computer" is a solution.

[Timoyoungster]

Sorry, thought it was about shifted characters like ':' or '{'.
Just to clarify: alt+tab doesn't work for me anymore (as to be expected xD), but in my use-case it's ok since needing pure luck to get into command mode in Vim or to save a file disrupts my workflow much more.

[yzhuang]

I studied this problem using Karabiner-Elements EventViewer.
I have a key for &kp LG(TAB) for quickly switching windows. When I press it, the OS receives the sequence shown in the attached image.

The bug is not about having no delay, but that the tab key down event is sent before the modifier key down events.
OS X handles this fine and my key works as I expect, however, many apps do not handle this sequence correctly and hence this buggy behavior.

This incorrect sequence happens consistently across all my other hotkey as well, such as &kp LG(LS(LC(NUM4))) (for taking screenshot on os x):

[kevinpastor]

Just tested on my side with RDP via Bluetooth and the CONFIG_ZMK_HID_SEPARATE_MOD_RELEASE_REPORT flag that #2334 brings turned on and the problem still remains.

[Timoyoungster]

Can you check whether it's actually sending separate reports?
(e.g. check whether a tool like wev shows the key events in the correct order)

Also, could you elaborate on how you got those results? Maybe a similar solution to #2334 in the keypress function could further improve the situation, but I can only test that if I can reproduce this. (For me the keys register correctly in all programs I've tried since #2334)

[kevinpastor]

I'm on Windows on the remote and the host, so the only software I know of for timing is the QMK Configurator test page.

This is what I get most of the time on Bluetooth and sometimes wired for &kp EXCL or &kp LS(N1):

KEY-DOWN - QMK: KC_LSFT Event key: Shift       Code: ShiftLeft     KeyCode: 16
KEY-UP   - QMK: KC_LSFT Event key: Shift       Code: ShiftLeft     KeyCode: 16 in 4.700ms
KEY-DOWN - QMK: KC_1    Event key: 1           Code: Digit1        KeyCode: 49
KEY-UP   - QMK: KC_1    Event key: 1           Code: Digit1        KeyCode: 49 in 121.300ms
KEY-UP   - QMK: KC_LSFT Event key: Shift       Code: ShiftLeft     KeyCode: 16 in 133.000ms

This is what I get otherwise:

KEY-DOWN - QMK: KC_LSFT Event key: Shift       Code: ShiftLeft     KeyCode: 16
KEY-DOWN - QMK: KC_1    Event key: !           Code: Digit1        KeyCode: 49
KEY-UP   - QMK: KC_1    Event key: !           Code: Digit1        KeyCode: 49 in 143.300ms
KEY-UP   - QMK: KC_LSFT Event key: Shift       Code: ShiftLeft     KeyCode: 16 in 152.000ms

For something like the right ALT modifier, it gets worst:

KEY-DOWN - QMK: KC_LCTL Event key: Control     Code: ControlLeft   KeyCode: 17
KEY-DOWN - QMK: KC_RALT Event key: AltGraph    Code: AltRight      KeyCode: 18
KEY-UP   - QMK: KC_TAB  Event key: Tab         Code: Tab           KeyCode: 9 in NaNms
KEY-UP   - QMK: KC_LCTL Event key: Control     Code: ControlLeft   KeyCode: 17 in 13.100ms
KEY-UP   - QMK: KC_RALT Event key: AltGraph    Code: AltRight      KeyCode: 18 in 11.300ms
KEY-DOWN - QMK: KC_2    Event key: 2           Code: Digit2        KeyCode: 50
KEY-UP   - QMK: KC_2    Event key: 2           Code: Digit2        KeyCode: 50 in 134.700ms
KEY-UP   - QMK: KC_LCTL Event key: Control     Code: ControlLeft   KeyCode: 17 in 158.400ms
KEY-UP   - QMK: KC_RALT Event key: AltGraph    Code: AltRight      KeyCode: 18 in 156.900ms

I have no idea how can these events be so much out of order. As I said in this thread, I have a macro setup to lengthen the timing of everything, but even with that I still get the issue here and there.

[Timoyoungster]

Weird ... tbh I don't really have an immediate idea as to how this could occur. I mean in the third example you aren't even pressing control right?
Could I maybe as you to share your config options?

[kevinpastor]

About the third example, I think it might be related to the fact that Alt Gr (right Alt) is sometimes interpreted as left Ctrl + left Alt. I'm unsure if this changes something, but my Windows keyboard layout is set to Canadian French.

There is a lot on my config, but the examples above have been generated with the keys on the left home keys: https://github.com/kevinpastor/zmk-config/blob/5d2609eb8db9e842378362d9a538656c7f6ca355/config/corne.keymap#L139

[Timoyoungster]

So it seems as if the mod releases (the ones after the actual keypress -> #2334) are done correctly.
Maybe starting by examining how the LS(), RA(), ... macros are (mis)handled in the keypress part could give some explanations for this behavior.
I'll see if I can reproduce this issue using those macros.

[kevinpastor]

Let me know if there's anything I can do to help or test.

[kevinpastor]

I just realized that caps word is also affected by this making it unusable on RDP 😕