[Bug] Inconsistent results when using send_unicode_string

[ScuttleSE]

Describe the Bug

I've been using an idobai id80 ISO for little more than a year without any issues at all. Recently I recompiled the keyboard firmware with the latest QMK, last compile was several months ago.

I have a few unicode-macros set up, that are no longer working properly.
One of them is supposed to output "(╯°□°）╯︵ ┻━┻"

If I hit the macro key ten times, I get this output:

(256f °25a1 °ff09 ╯fe35  ┻2501 ┻)
(256f °25a1 °ff09 ╯fe35  253b ━253b )
(256f °25a1 °ff09 ╯fe35  253b ━253b )
(256f °25a1 °ff09 ╯fe35  253b ━253b )
25(6f 25°a1 °）256f ︵0020 ┻2501 ┻)
2(56f °25a1 °ff09 ╯fe35 25 3b ━253b )
(256f °25a1 f°f09 ╯0020 ┻2501 ┻)
25(6f °25a1 ff°09 fe╯35 25 3b ━┻)
(256f °25a1 °ff09 ╯fe35  253b ━┻)
(╯°00b0 ）256f 0020 ┻━┻)

As you can see, it's not consistent. There are some unicode characters that are entered correctly sometimes, and Ctrl+Shift+U and then manually entering the code works just fine.

This is how I have built the macro:

enum custom_keycodes {
    JAP_SHRUG = SAFE_RANGE
};

bool process_record_user(uint16_t keycode, keyrecord_t *record) {
    switch (keycode) {
    case JAP_SHRUG:
        if (record->event.pressed) {
            send_unicode_string("(╯°□°）╯︵ ┻━┻");
        } else {
            // when keycode is released
        }
        break;

I have then bound JAP_SHRUG to a keyboard key.

System Information

Keyboard: Idobao id80 ISO
Revision (if applicable): v1
Operating system: Arch Linux
qmk doctor output:

Ψ QMK Doctor is checking your environment.
Ψ CLI version: 1.0.0
Ψ QMK home: /home/scuttle/qmk_firmware
Ψ Detected Linux.
Ψ Git branch: master
Ψ Repo version: 0.16.9
Ψ All dependencies are installed.
Ψ Found arm-none-eabi-gcc version 11.3.0
Ψ Found avr-gcc version 8.3.0
Ψ Found avrdude version 6.4
Ψ Found dfu-util version 0.11
Ψ Found dfu-programmer version 0.7.2
Ψ Submodules are up to date.
Ψ QMK is ready to go

Any keyboard related software installed?

• AutoHotKey (Windows)
• Karabiner (macOS)
• Other:

Additional Context

[ScuttleSE]

Hm, the more I look into this, the less sure I am about the issue being QMK... if I kill ibus, I can run the macro without any issues whatsoever... Problem is that it stops working in Telegram f.ex....

[ScuttleSE]

Okay...after a lot of fiddling around I have come to the conclusion that send_unicode_string is too fast for some reason.

If I send a unicode string like this, I get one or two characters right, the rest will give me garbage output:

    case JAP_TABLEREST:
        if (record->event.pressed) {
            send_unicode_string("┬─┬ノ(ಠ_ಠノ)");
        } else {
            // when keycode is released
        }

If I do this, it works just fine:

    case JAP_TABLEREST:
        if (record->event.pressed) {
            send_unicode_string("┬");
            _delay_ms(30);
            send_unicode_string("─");
            _delay_ms(30);
            send_unicode_string("┬");
            _delay_ms(30);
            send_unicode_string("ノ");
            _delay_ms(30);
            send_unicode_string("(");
            _delay_ms(30);
            send_unicode_string("ಠ");
            _delay_ms(30);
            send_unicode_string("_");
            _delay_ms(30);
            send_unicode_string("ಠ");
            _delay_ms(30);
            send_unicode_string("ノ");
            _delay_ms(30);
            send_unicode_string(")");
            _delay_ms(30);
        } else {
            // when keycode is released
        }

Is there a way to have a user definable wait-period between each character in send_unicode_string?

[vinorodrigues]

Let's look at the basics first.

1. your source files are saved in UTF-8 ? (Not sure what editor you use, but if it supports EditorConfig you want to add charset = utf-8 to the .editorconfig file.)
2. Your rules.mk file has UNICODE_ENABLE = yes right? (because the repo files for the ID80-ISO does not, so you need it in your keymap rules.mk.)

If yes to all above then you can try setting the Input Modes to your platform, see: https://docs.qmk.fm/#/feature_unicode?id=input-modes (actually have a good read of that whole page as there is some info on toggling multiple input modes etc.)

[ScuttleSE]

Yes, yes and yes :)

The problem isn't getting unicode to work, the problem seems to be the ibus daemon getting confused when the macro is being typed too fast.

If you look at the two code examples, they do the exact same thing, but the second one does it about one third of a second slower

[vinorodrigues]

umm... code does have delays in it: https://github.com/qmk/qmk_firmware/blob/master/quantum/process_keycode/process_unicode_common.c

1. send_unicode_string decodes the utf-8 and then calls register_unicode.
2. register_unicode calls unicode_input_start , then loops the string, then calls unicode_input_finish
3. now looking at unicode_input_start there is line 122 wait_ms(UNICODE_TYPE_DELAY);

not sure of why the wait is in there ... but try this as an experiment ... remove the _delay_ms(30); lines in your 2nd set but keep the separate send_unicode_string.

[vinorodrigues]

Also #15061 states that on Windows a wait needs to be inserted... but that PR addresses a single character KC_PPLS. ... maybe there is something to explore. Are you on Windows on this one?

[ScuttleSE]

Oh, no, Linux

Interesting...never knew about the UNICODE_TYPE_DELAY setting though...

[ScuttleSE]

Okay...tried a few different versions... Removed the _delay_ms(30); lines, but kept the string as separate send_unicode_string lines. No luck, getting garbled output similar to the one in my first post

Same with adding #define UNICODE_TYPE_DELAY 30 in my config.h

[daishan]

I had the same issue. Increasing UNICODE_TYPE_DELAY didn't help, but introducing two additional delays right before and after finishing the input of a Unicode character did fix the issue for me.

See this commit: 9f62e26

[rubienr]

I suggest another, less invasive, fix by adding the delay to unicode.c in unicode_input_start() only once (before) tap_code16(UNICODE_KEY_LNX);:

unicode.c:

case UNICODE_MODE_LINUX:
    wait_ms(UNICODE_TYPE_DELAY); // additional delay
    tap_code16(UNICODE_KEY_LNX);
    break;

Then eventually increase the delay in config.h: #define UNICODE_TYPE_DELAY 20. With a STM32L432 board it was definitively necessary.

[rubienr]

And so the circle is complete!

It took me a day to debug, then find this post and to realize that i already replied once 🤦!
My mitigation strategy is:

• restarting ibus: ibus restart
• adding a more delays

Relates to PR #22353