How to begin

[Martoni]

Hello,

Is it possible to have a little how-to use this plugin ?

Thank you.

[edmundmtang]

Heya! Thank you for your interest in this project of mine!

First off, this is one of the first things I've ever put out there, so I am not entirely sure what I'd need to provide in order to help you use this plugin. Secondly, I didn't build this with widespread compatibility in mind, so as far as I know this might only work on Windows. Thirdly, it's been a while since I looked at this project so bear with me while I gather my thoughts and figure out how to clearly communicate what is needed to get this to work.

To begin with, I guess I'd start with asking which version of Godot you want to use this in? 4.0? or 4.1?

[Martoni]

Hey,
Thanks for this fast response. I'm working on Linux, I already tryied Godot 3.2.3 to make a space-invader like game. But I think I can migrate the project on godot 4.0 (and publish it on github soon).
I already tested Vosk with python to make a learning game for calculation (in french): https://github.com/Martoni/FaisTesDevoirsAvant/blob/master/python/vosk/multvosk.py

My goal is to make a space invader like game but with some question to respond right by voice to reload the weapon ;)

[edmundmtang]

I just pushed an update in godot_4.1.1 that updates the project to Godot 4.1.1, fixes references to submodules, and includes the required binaries. If you're lucky, you just need to do two things to get the project to run.

1. First, you'd obviously need to clone the repo. The main branch is currently on 4.1.1 stable, so you can just run git clone git@github.com:edmundmtang/vosk-godot.
2. Second, you'd need to put in an appropriate vosk language model, which you can download from here. Once downloaded, unzip the language model in project/speech_recognizer/language_models/. I've been using vosk-model-small-en-us-0.15 in my own projects, and that is the default in the project, so if you decide to use a different LM then you'll also have to update the LM path in the GDScript.
3. Once you've done these things, the project should run, though you might need to adjust some settings like the audio mix rate to match your system. However, since you mentioned that you're working on Linux you might need to recompile all the binaries in order to get the project to run (I'm not to familiar with cross-platform compatibility, so I'm just guessing from here out).

If you need to recompile the binaries, there are up to 3 sets of binary files that may need to be recompiled.

1. The first is my fork of the Vosk API. I'm actually just going to ignore this one for now because it's a different beast that I hope we won't need to address. Also, I compiled it on a WSL so I'm hoping it won't present a compatibility issue. If everything else fails we might look into this. In any case, the binaries from compiling the Vosk API are libpath/libvosk.lib and project/bin/libvosk.dll. Note that there are 3 dependencies for libvosk that are also included in the project/bin/ directory (libgcc_s_seh-1.dll, libstdc++-6.dll, libwinpthread-1.dll). We don't need to ever recompile these; we just take them from Vosk.
2. The first thing that I'd actually try to recompile is the godot-cpp/ binaries. To begin, you'd need to fetch the files, so in the main directory of the repo run:
   git submodule init
git submodule update --recursive
   Then navigate to the submodule directory using cd godot-cpp. You can then compile the CPP bindings using the command scons. It should figure out your system settings, but you may need to pass additional arguments like platform=linux.
3. With godot-cpp and Vosk files, you can then go back to the main directory and compile the vosk-godot binaries by calling upon scons again. That should generate the appropriate binaries and put them in the project/bin folder. At this point, make sure you have an appropriate language model in the correct directory and it should hopefully work.

[edmundmtang]

Before you get too deep into making this work, I would point out this issue that I found with the Godot engine. My current workaround is to periodically restart the microphone from inside the engine. I don't think it's a deal-breaker, but I think it's worth noting.

[Martoni]

Thanks for the help, I'll try it.
I just uploaded the upgraded project BrainDefender on github. I hope I'll find time this weekend to test vosk.

[Martoni]

I got some time to test the extension installation but, of course, I have to recompile it for Linux :

Error loading extension : res://speech_recognizer/vosk_speech_recognizer.gdextension

I'm trying to recompile it with scons without success for the moment. I'm stuck in compiling vosk library in linux.
All tests I've done so far with vosk was with Python.

[edmundmtang]

Thanks for the feedback. Unfortunately it's was sorta expected that it wouldn't be so easy. Looking back upon what I wrote, if we need to recompile the Vosk API for Linux we'd be looking for different files from the ones I mentioned (I said .dll, but that's not what we need on Linux).

What issues are you running into when compiling with scons? I'll try to look into compiling the libraries for Linux.

[Martoni]

Finally, I managed to compile library with scon. I had a problem with -lvosk option solved by unziping vosk-api release so file in /usr/local/lib/ :

$ sudo cp vosk-linux-x86_64-0.3.45/libvosk.so /usr/local/lib/

Then launching scons :

$ scons platform="linux"
scons: Reading SConscript files ...
Auto-detected 8 CPU cores available for build parallelism. Using 7 cores by default. You can override it with the -j argument.
Building for architecture x86_64 on platform linux
scons: done reading SConscript files.
scons: Building targets ...
scons: `godot-cpp/bin/libgodot-cpp.linux.template_debug.x86_64.a' is up to date.
g++ -o project/bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so -Wl,-R,'$ORIGIN' -m64 -march=x86-64 -shared src/register_types.os src/vosk_speech_recognizer.os -Lgodot-cpp/bin -Llibpath -lgodot-cpp.linux.template_debug.x86_64 -lvosk
scons: done building targets.

I modified file project/speech_recognizer/vosk_speech_recognizer.gdextension to add linux library version :

[configuration]

entry_symbol = "vosk_speech_recognizer_library_init"
compatibility_minimum = 4.1

[libraries]

windows.debug.x86_64 = "res://bin/lib_vosk_speech_recognizer.windows.template_debug.x86_64.dll"
linux.debug.x86_64 = "res://bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so"

Named according to binary file located in :

project/bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so

But I've still got the extension error message when I'm loading project with godot :

Error loading extension : res://speech_recognizer/vosk_speech_recognizer.gdextension

Main error message is :

Parser Error: Could not find base class "VoskSpeechRecognizer".

[edmundmtang]

One thing to note about gdextension in its current form is that its error messages are actually quite uninformative. It'll give this same error for a number of different issues. Since you used the .so from the standard vosk distribution I think that may be the issue. I'll look into recompiling my modified version of vosk so that we have a library for Linux use.

[edmundmtang]

I tried recompiling the binaries and putting them in a new branch linux_compatibility. I still need to test it, but I need to figure out how to get Godot on Linux working to test it.

[Martoni]

I copied the libvosk.so file in project/bin and godot managed to find the library apparently.
I've still have some error in console :

$  
ERROR: Cannot get class 'AudioEffectCaptureExtend'.
   at: instantiate (core/object/class_db.cpp:339)
ERROR: res://default_bus_layout.tres:3 - Parse Error: Can't create sub resource of type: AudioEffectCaptureExtend
   at: load (scene/resources/resource_format_text.cpp:550)
ERROR: Failed loading resource: res://default_bus_layout.tres. Make sure resources have been imported by opening the project in the editor at least once.
   at: _load (core/io/resource_loader.cpp:273)
WARNING: Blend file import is enabled in the project settings, but no Blender path is configured in the editor settings. Blend files will not be imported.
     at: _editor_init (modules/gltf/register_types.cpp:73)
ERROR: Can't open dynamic library: /opt/vosk-godot/project/bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so. Error: /opt/vosk-godot/project/bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so: undefined symbol: vosk_recognizer_set_timings.
   at: open_dynamic_library (drivers/unix/os_unix.cpp:650)
ERROR: GDExtension dynamic library not found: /opt/vosk-godot/project/bin/lib_vosk_speech_recognizer.linux.template_debug.x86_64.so
   at: open_library (core/extension/gdextension.cpp:455)
ERROR: Failed loading resource: res://speech_recognizer/vosk_speech_recognizer.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: _load (core/io/resource_loader.cpp:273)
SCRIPT ERROR: Parse Error: Could not find base class "VoskSpeechRecognizer".
          at: GDScript::reload (res://speech_recognizer/VoskSpeechRecognizer.gd:1)
ERROR: Cannot get class 'VoskSpeechRecognizer'.
   at: instantiate (core/object/class_db.cpp:339)

[edmundmtang]

You copied libvosk.so, but did you also recompile lib_vosk_speech_recognizer.linux.template_debug.x86_64.so or use the new file from the new branch? I still haven't gotten around to getting Godot in a VM so I haven't tested it myself yet.

[edmundmtang]

Upon further inspection, I am likely compiling the libvosk.so file incorrectly so I will be looking into recompiling it.

[edmundmtang]

I made another push to linux_compatibility. I was able to test whether or not it runs in an Ubuntu VM and I was able to get it to run without errors. Unfortunately, I haven't set up microphone access in my VM yet, so I haven't been able to test its full capabilities. Please let me know if it works now!

I'll figure it out how to test it with microphone access later, if it doesn't work or if you don't get around to testing it.

P.S. - Both the libvosk.so and lib_vosk_speech_recognizer.linux.template_debug.x86_64.so files are replaced in this version.