Help needed

[PhilippeRo]

Hi,

I'm developping an ibus engine (called "stt") and I experienced something strange.

Every time I click on the name of the engine in the ibus engine menu in the top gnome-shell bar (see below the picture), it creates a new "stt" engine, EVEN IF the "stt" engine is already selected. One of the symptoms among others is that the new engine looses the session parameters of the formerly active one and it stops voice recognition.

I tried to confirm this with your engine that is far more advanced and complex than mine. I turned on debug, launched it from the command line, then I clicked on its name in the menu, it started, and, I clicked several times on its name in the menu again. I read the debug log and I got the following lines:
2022-06-24 18:03:21,070 factory.py line 64 do_create_engine DEBUG: EngineFactory.do_create_engine(engine_name=typing-booster)
2022-06-24 18:03:28,135 factory.py line 64 do_create_engine DEBUG: EngineFactory.do_create_engine(engine_name=typing-booster)
2022-06-24 18:03:31,350 factory.py line 64 do_create_engine DEBUG: EngineFactory.do_create_engine(engine_name=typing-booster)
2022-06-24 18:03:40,751 factory.py line 64 do_create_engine DEBUG: EngineFactory.do_create_engine(engine_name=typing-booster)
2022-06-24 18:03:45,565 factory.py line 64 do_create_engine DEBUG: EngineFactory.do_create_engine(engine_name=typing-booster)

Could you confirm a new engine is indeed created every time I clicked though typing-booster was already selected?
Does it seem normal to you?

I'd like to add that there is a second problem connected to this one, at least for my engine, which is ibus does not destroy the engines created before and they seem to lie around doing nothing. I know this because some of the objects they created still receive signal notifications.
I know this issue seems to have to do with ibus and not your engine but I wanted to your opinion on this.

I attached the full log.
debug.log

Thank you.

[mike-fabian]

Could you confirm a new engine is indeed created every time I clicked though typing-booster was already selected? Does it seem normal to you?

Yes, I think that is normal.

The reason is probably that some engines (like ibus-table which I also maintain) can can be started with different modes to create fairly different engine.

There are many “table:...” engines in case of ibus table: for example there is the engine “table:cangjie5” for Chinese input using cangjie5 and there is the “table:latex” to input LaTeX symbols. These use completely different tables and thus behave very differently. But they all use the same ibus-table code, they are just started with different names.

But of course one does not want to exactly the same engine twice!

So I check whether the exact same engine (same engine_name) is already running and if yes I reuse it instead of creating a new one. And also reuse its database instead of creating a new database connection.

This code in factory.py does it:

https://github.com/mike-fabian/ibus-typing-booster/blob/main/engine/factory.py#L70

        try:
            if engine_name in self.database_dict:
                self.database = self.database_dict[engine_name]
            else:
                self.database = tabsqlitedb.TabSqliteDb()
                self.database_dict[engine_name] = self.database
            if engine_name in self.enginedict:
                engine = self.enginedict[engine_name]
            else:
                engine = hunspell_table.TypingBoosterEngine(
                    self.bus, engine_path + str(self.engine_id), self.database)
                self.enginedict[engine_name] = engine
                self.engine_id += 1
            return engine
        except Exception as error:
            print(f'failed to create engine {engine_name}: '
                  f'{error.__class__.__name__}: {error}')
            import traceback
            traceback.print_exc()
            raise Exception from error

In case of Typing Booster there is currently only one engine_name typing-booster, not many possibilities like for ibus-table. So currently self.enginedict will hold either 0 or one engines, never more than 1 at the moment. Same for self.database_dict, it will be either empty or have one database at the moment. And self.engine_id will be either 0 or one, never greater than 1.

But I have some ideas of adding some special purpose Typing Booster engines in future, maybe. So this may change and there may be special Typing Booster engines in future. For example Typing Booster can emulate the behaviour of ibus-m17n when setup in a special way. See the chapter “Simulate the behaviour of ibus-m17n” in the online documentation:

https://mike-fabian.github.io/ibus-typing-booster/docs/user/#2_2_1_1

So maybe I will introduce additional special engine names in future, maybe something like typing-booster-m17n:or:inscript2 which would then be automatically setup in a way to emulate the behaviour of the or-inscript2 engine of ibus-m17n.

[mike-fabian]

By the way, I see that you are developing a speech-to-text engine.

Which speech-to-text software are you intending on using?

I ask because ibus-typing-booster also has a speech-to-text feature using the service from the Google cloud.

This service has a good quality, but it quite hard to setup. See the chapter “Speech recognition” in the online-documentation:

https://mike-fabian.github.io/ibus-typing-booster/docs/user/#10

Unfortunately this Google service is quite difficult to setup and it is only free of charge for a limited time per month. Something like 1 hour per month is free, above that one has to pay.

What service are you using?

If there are good alternatives to the Google service, I would like to add options to choose different services.

[PhilippeRo]

Thanks a lot for your reply. I will dig into that. The most worrying thing to me is if the previous engine is not properly destroyed, but since it does not happen for typing-booster, the error is mine.
To answer your last question, I did see that your engine had a speech to text feature but I wanted something local when I started and discovered vosk which works quite well (granted not as good as google). So it's not an online service though it has been designed to work on servers to provide an online service.
I haven't put the code online but it should reach a usable state in the coming month(s), if all goes well.

[PhilippeRo]

Hi again,
sorry to bother you with one more question but I thought you might help me since you know a lot about IBus internals and the documentation is sparse or a bit outdated sometimes. I tried to send this question on the two forums (the google ones) or in the online chat forum with no answer.
I saw you commented recently on the function forward_key_events(). Do you know if it is possible to send a key combination to the client application, something such as "Ctrl + Z" to trigger client's shortcuts ? I tried, hard, but I could not manage to do this and it would be helpful to send "commands".
Thanks a lot.

[mike-fabian]

I saw you commented recently on the function forward_key_events(). Do you know if it is possible to send a key combination to the client application, something such as "Ctrl + Z" to trigger client's shortcuts ? I tried, hard, but I could not manage to do this and it would be helpful to send "commands". Thanks a lot.

Yes, that is possible and I am doing it in ibus-typing-booster also.

self.forward_key_event(IBus.KEY_Z, 0, IBus.ModifierType.CONTROL_MASK)

should forward a Ctrl+Z.

[mike-fabian]

In Typing Booster, I have this helper function

    def _forward_generated_key_event(self, keyval: int) -> None:
        '''Forward a generated key event for keyval to the application.'''
        # Without using a correct ibus key code, this does not work
        # correctly, i.e. something like
        # self.forward_key_event(IBus.KEY_Left, 0, 0) used to work
        # (once upon a time) but it does *not* work anymore now!
        #
        # The ibus key code for IBus.KEY_Left is usually 105, but it
        # could be different on an unusual keyboard layout.  So do not
        # hardcode keycodes here, calculate them correctly for the
        # current layout.
        keycode = self._keyvals_to_keycodes.keycode(keyval)
        ibus_keycode = self._keyvals_to_keycodes.ibus_keycode(keyval)
        keystate = 0
        if DEBUG_LEVEL > 0:
            LOGGER.debug('keyval=%s keycode=%s ibus_keycode=%s keystate=%s',
                         keyval, keycode, ibus_keycode, keystate)
        self.forward_key_event(keyval, ibus_keycode, keystate)

which I use to forward generated Left key events which didn’t really happen (I use that to adjust the cursor position), for example:

            if not candidate_was_selected:
                # cursor needs to be corrected leftwards:
                for dummy_char in commit_string[caret_was:]:
                    self._forward_generated_key_event(IBus.KEY_Left)

Here I have keystate = 0 always but if I needed Control, I could easily extend that function to add a state parameter as well.

(I currently pass stuff like Ctrl+Left also to the application using forward_key_event() but only when this key event actually happened. In that case I already have the all three values key.val, key.code, and key.state and don’t need that helper function, just call

self.forward_key_event(key.val, key.code, key.state)

I have this comment in the helper function which says that using 0 always for keycode used to work once upon a time, but it stopped working long ago and I had to calculate the real keycode. So from that helper function I call another helper function which calculates the ibus_keycode from keyval (Depends on the current keyboard layout).

But I don’t think you need to do that anymore, recent versions of ibus calculate the right keycode automatically if you pass in 0 for keycode.

I.e. with a recent enough ibus, just doing

self.forward_key_event(IBus.KEY_Z, 0, IBus.ModifierType.CONTROL_MASK)

with a 0 for the keycode should work to forward a Ctrl+Z.

[mike-fabian]

Warning: forward_key_event() does not currently work when the Gtk4 input module is used! I am not sure when or if this will be fixed.

Because of this problem (and because forward_key_event() didn’t work in very old versions of Qt), Typing Booster has an option

☑️ Avoid using the forward_key_event() function

If that option is on (it is off by default), I try to avoid forward_key_event() when possible and use return False instead. That looses some nice features though, I can only use return False as a replacement when I am forwarding an event I actually received, not for generated key events like those I need to adjust the cursor position. On top of that it causes ordering problems in XIM programs like xterm: When using return False, typing test + space will result in test being committed in xterm instead of the correct test .

[mike-fabian]

In the newest code I try to detect whether forward_key_event() is usable or not automatically, see:

https://github.com/mike-fabian/ibus-typing-booster/blob/main/engine/hunspell_table.py#L6019

 if (self._avoid_forward_key_event
                or
                (hasattr(IBus.Capabilite, 'SYNC_PROCESS_KEY')
                 and
                 self.client_capabilities & IBus.Capabilite.SYNC_PROCESS_KEY)):
                return self._return_false(key.val, key.code, key.state)
            self.forward_key_event(key.val, key.code, key.state)
            return True

ibus >= 1.5.27 will have this new IBus.Capabilite.SYNC_PROCESS_KEY attribute, ibus < 1.5.27 does not have that. I don’t check the ibus version, I just check whether this attribute is available. If it is available and this flag is set in self.client.capabilities, then this means that forward_key_event() does not work. This is currently the case only in the Gtk4 input module.

With the above code, if ibus is new enough, the workaround to avoid forward_key_event() is automatically applied for Gtk4 only without the need to use the ☑️ Avoid using the forward_key_event() function workaround option which loses features and even causes severe problems for other applications

[PhilippeRo]

Thank you very much for all your tips and your guidance.
I tried several things but to no avail, probably because with fedora 36 most applications use GTK4 and you warned me about this.
At best I get "z" characters and if I hold Ctrl at the same time, that works. Only Gimp seems to work but I think it must use gtk2.
Another option I'm considering is https://gitlab.freedesktop.org/libinput/libei but it has no bindings for the time being and is not packaged yet (except for fedora 35) so it can only be a long term solution.
Thanks again.

[mike-fabian]

I tried as well what you did and did:

diff --git a/engine/hunspell_table.py b/engine/hunspell_table.py
index 7f68b3f4..3f39b3fb 100644
--- a/engine/hunspell_table.py
+++ b/engine/hunspell_table.py
@@ -5628,7 +5628,11 @@ class TypingBoosterEngine(IBus.Engine): # type: ignore
         key = itb_util.KeyEvent(keyval, keycode, state)
         if DEBUG_LEVEL > 1:
             LOGGER.debug('KeyEvent object: %s', key)
-
+        self.forward_key_event(
+            IBus.KEY_Left,
+            self._keyvals_to_keycodes.ibus_keycode(IBus.KEY_Left),
+            IBus.ModifierType.CONTROL_MASK)
+        return True
         if self._handle_compose(key):
             return True
 

and the same with the 0 instead of calculating the key code:

diff --git a/engine/hunspell_table.py b/engine/hunspell_table.py
index 7f68b3f4..7b39cb81 100644
--- a/engine/hunspell_table.py
+++ b/engine/hunspell_table.py
@@ -5628,7 +5628,11 @@ class TypingBoosterEngine(IBus.Engine): # type: ignore
         key = itb_util.KeyEvent(keyval, keycode, state)
         if DEBUG_LEVEL > 1:
             LOGGER.debug('KeyEvent object: %s', key)
-
+        self.forward_key_event(
+            IBus.KEY_Left,
+            0,
+            IBus.ModifierType.CONTROL_MASK)
+        return True
         if self._handle_compose(key):
             return True
 

and found that the version with the 0 does not work!

I thought ibus had fixed this recently. So it looks like I have to keep my own key code calculation for another while ...

I also tried

diff --git a/engine/hunspell_table.py b/engine/hunspell_table.py
index 7f68b3f4..21c020d4 100644
--- a/engine/hunspell_table.py
+++ b/engine/hunspell_table.py
@@ -5628,7 +5628,11 @@ class TypingBoosterEngine(IBus.Engine): # type: ignore
         key = itb_util.KeyEvent(keyval, keycode, state)
         if DEBUG_LEVEL > 1:
             LOGGER.debug('KeyEvent object: %s', key)
-
+        self.forward_key_event(
+            IBus.KEY_z,
+            self._keyvals_to_keycodes.ibus_keycode(IBus.KEY_z),
+            IBus.ModifierType.CONTROL_MASK)
+        return True
         if self._handle_compose(key):
             return True
 

and this also works only when I use my key code calculation, not with the 0.

Note that these simple tests send the Control+Left or Control+z once for pressing a key and another time for releasing a key. I.e. if I do the Control+Left test I see the cursor skip over two words in gedit when I press and release for example the a key ...

[PhilippeRo]

I used your above patches (the two that use your function, since passing 0 fails for you too) but they don't work for me.
I get a double "z" and the cursor moves left, but by two characters, not two words as expected. Note that I had to turn off character repetition since otherwise z or left arrow get printed over and over again as the client app never receives a release event.

I wanted to know the values received through ibus for the two key combinations (Ctrl + z / left) when you press them, so I added print("The values", keyval, keycode, self._keyvals_to_keycodes.ibus_keycode(IBus.KEY_z/Left) before everything else. Then I started ibus-typing-booster and pressed "ctrl z" / "ctrl Left".
I get (122, 17, 17) for "ctrl z" and (65361, 105, 105) for "ctrl Left". 105 is also the number that is indicated in your comment above.
So, your function seems to be right about the keycode but something else is wrong in my case.

I'm starting to wonder if your code that moves the cursor left by two words to correct its position would work for me. I could check that if I knew how to trigger it.

For the record, I'm using ibus 1.5.26 and typing-booser 2.16.7.
My keyboard is azerty.

[mike-fabian]

Note that I had to turn off character repetition since otherwise z or left arrow get printed over and over again as the client app never receives a release event.

Press event:

        self.forward_key_event(
            IBus.KEY_Left,
            self._keyvals_to_keycodes.ibus_keycode(IBus.KEY_Left),
            IBus.ModifierType.CONTROL_MASK)

Release event:

        self.forward_key_event(
            IBus.KEY_Left,
            self._keyvals_to_keycodes.ibus_keycode(IBus.KEY_Left),
            IBus.ModifierType.CONTROL_MASK|IBus.ModifierType.RELEASE_MASK)

But I never forward release events at the moment, that does not seem to be necessary.

[mike-fabian]

I'm starting to wonder if your code that moves the cursor left by two words to correct its position would work for me. I could check that if I knew how to trigger it.

Does this work for you?:

With ibus-typing-booster, type test then Left twice so that the cursor is after the te in the middle of the preedit. Then Return.
Now the result should be

te
|st

where | indicates the cursor position.

When the Return is typed, the part to the left of the cursor, i.e. te is committed first. Then the part to the right of the cursor, i.e. st is committed. After this second commit, the result is

test|

because after a commit the cursor is always to the right of the commit (In this simple case of ASCII, I wouldn’t have needed the two commits, I could have committed test in one commit, but if it is not ASCII and some transliteration is ongoing, committing the part to the left and to the right of the cursor can make a difference).

After the second commit, two artificial Left key events are forwarded to the application to get this:

te|st

And then finally, the Return key event which triggered the commit is forwarded so one gets the final result:

te
|st

Does this work for you?

Works for me in gedit in a Gnome Xorg session, see this video:

Peek.2022-07-03.22-10.mp4

[PhilippeRo]

As far as I can tell, yes what you described works as expected. The cursor is moved left by two characters.
It goes so fast that I made a video like you to record the moves of the cursor.

2022-07-05.10-55-56.mp4

I should mention that I used obs and that it did not record the typing-booster menu with the suggestions for completion but it was there (but I chose single-window mode in obs).

[PhilippeRo]

Sorry to dig up this issue again but I finally figured out what was going on.
On wayland (and wayland only), for gtk3 and gtk4, the client im-module used is NOT ibus but GTK's own client (usually compiled and linked inside the library, at least fedora does). And this client does not allow key combination (I still need to see the code to check what's going on) contrary to ibus client.
If I set GTK_IM_MODULE to ibus (still in wayland) and launch gedit, ..., from the command line, then I get the behaviour that people get on Xorg (which must be yours).
Lesson learnt, im clients used by gtk, whether dbus' or its own, are behaving quite differently (and I've seen other areas where they differ).

Thanks again for your help.