Google cloud speech to text fails with exception : Unavailable

[sychopath]

I am trying to use google-cloud-cpp speech to text apis. I am able to run sample codes using as a standalone application by building a shared object(.so) file have google stt code and calling same functions through an executable.
When I am using the same set of library to call apis inside my application plugin(Unimrcp), I get the below exception(status):

UNAVAILABLE: Error in non-idempotent operation Recognize: failed to connect to all addresses; last error: UNKNOWN: Network is unreachable

System information : I am using a gcp server with Ubuntu 18.04.6 LTS. Built a sample app and unimrcp application using Makefiles.
I have saved google credentials through environment variable. Also, have verified that the request is not reaching my api server as there is no log.

There should not be any network issues as I am to reach apis through another executable. Moreover, I am using another set of plugin with my Unimrcp service for azure stt, which works fine.

I have also captured data packets using wireshark, a snapshot of that is attached.

[coryan]

I am trying to use google-cloud-cpp speech to text apis.

Do you know what version you are using? Likewise for gRPC, protobuf, Abseil, and OpenSSL?

When I am using the same set of library to call apis inside my application plugin(Unimrcp), I get the below exception(status):

UNAVAILABLE: Error in non-idempotent operation Recognize: failed to connect to all addresses; last error: UNKNOWN: Network is unreachable

Is it possible to run this code with the following environment variables? That will add some logging that may help: GOOGLE_CLOUD_CPP_ENABLE_CLOG=yes and GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc,rpc-streams.

Also, have verified that the request is not reaching my api server as there is no log.

I am not sure what is your API server. Nor what request you are referring to. But I take it that this is an indication that the TTS request failed.

There should not be any network issues as I am to reach apis through another executable. Moreover, I am using another set of plugin with my Unimrcp service for azure stt, which works fine.

When you use google-cloud-cpp as part of the plugin: is this plugin dynamically loaded?

[coryan]

Thank you for your suggestions. I have added grpc environment variables .

Wow, that is a lot of logging. Sorry you had to wade through that much.

I have attached the grpc logs from console. I can see an error related to security handshake :

Security handshake failed: UNKNOWN:Peer name speech.googleapis.com is not in peer certificate

Not sure why the same set of code/library works fine at one place and fails at other.

I am not sure either. There are a couple of hypothesis that come to mind:

• The environment where you have deployed this does not have the ca-certificates package (or a similar name) installed. That sometimes happens when creating Docker images, but you said the Azure equivalent worked, so that seems unlikely.
• If the OpenSSL library is linked dynamically you may be using a different OpenSSL library in one environment vs. the other. It may be that the two OpenSSL versions behave differently, e.g., expect to find the root certificates in different locations. From your previous comments you are using vcpkg to compile google-cloud-cpp and its dependencies.
  • Can you let us know which triplet you used to compile the libraries?
  • Does that triplet produce shared libraries or static libraries?
• (unlikely) Maybe the environment where the plugin is loaded has limited access to the file system? Maybe via some kind of container or via some kind of LD_PRELOAD?

As a workaround (or maybe to test some of these hypothesis). You can try setting GRPC_DEFAULT_SSL_ROOTS_FILE_PATH and point this to a roots.pem file that you create yourself. For example, you can download https://pki.google.com/roots.pem and point the environment variable to it.

[sychopath]

The environment where you have deployed this does not have the ca-certificates package (or a similar name) installed. That sometimes happens when creating Docker images, but you said the Azure equivalent worked, so that seems unlikely.

I am loading my plugin inside a service. But there could be a possibility that my plugin is not able to access my environment variables.

If the OpenSSL library is linked dynamically you may be using a different OpenSSL library in one environment vs. the other.

I have fixed this by updating openssl to 3.0.5 version, had an older version installed on my system.

From your previous comments you are using vcpkg to compile google-cloud-cpp and its dependencies.
Can you let us know which triplet you used to compile the libraries?
Does that triplet produce shared libraries or static libraries?

I have installed/downloaded all the libraries through vcpkg. I see only static library files(.a) installed on my system. I have followed the same instruction mentioned on github.

https://github.com/googleapis/google-cloud-cpp
https://github.com/googleapis/google-cloud-cpp/tree/main/google/cloud/speech/quickstart

Maybe the environment where the plugin is loaded has limited access to the file system? Maybe via some kind of container or via some kind of LD_PRELOAD?

Not sure, but my plugin gets loaded inside a service or even from executable and I see no difference.

As a workaround (or maybe to test some of these hypothesis). You can try setting GRPC_DEFAULT_SSL_ROOTS_FILE_PATH and point this to a roots.pem file that you create yourself. For example, you can download https://pki.google.com/roots.pem and point the environment variable to it.

I downloaded the certificate from the link and have set the environment variable with its path but I don't see any differences in the result from standalone app or plugin execution.
I looked into the grpc logs from my working app(executable) to see the contents of certificate exchange but don't see the data referred by new pem file. I am new to grpc and not sure if I understood your point correctly.

Thanks again, Really appreciate your help here @coryan !!

[coryan]

I should start by saying that I am not sure what the problem is, and you should not put too much weight on what I say. I may be completely off-base and apologize if I waste your time.

I think at this point I really need to understand your link line to create the plugin .so file.

I am loading my plugin inside a service. But there could be a possibility that my plugin is not able to access my environment variables.

I do not recall if you mentioned this: do you use the same operating system and distribution to develop and deploy your service and the plugin? OpenSSL figures out where to find the root certificates at compile-time, but different distributions use different paths for the root certificates. If you were (for example) developing on a Fedora or RedHat system the root certificates are at /etc/pki/tls/certs/ca-bundle.crt, if you then deploy to an Ubuntu or Debian system they would be at /etc/ssl/certs/ca-certificates.crt. An OpenSSL version compiled on RedHat/Fedora could not find the certificates if deployed to Ubuntu or Debian.

If the OpenSSL library is linked dynamically you may be using a different OpenSSL library in one environment vs. the other.

I have fixed this by updating openssl to 3.0.5 version, had an older version installed on my system.

I take it that this did not fix the problem, you just made sure that inconsistent OpenSSL versions were not the problem. Did I read that right?

I have installed/downloaded all the libraries through vcpkg. I see only static library files(.a) installed on my system. I have followed the same instruction mentioned on github.

https://github.com/googleapis/google-cloud-cpp https://github.com/googleapis/google-cloud-cpp/tree/main/google/cloud/speech/quickstart

That all sounds good. Can you share how you build the plugin? In particular, how do you invoke the compiler (or linker) to create the plugin .so?

Maybe the environment where the plugin is loaded has limited access to the file system? Maybe via some kind of container or via some kind of LD_PRELOAD?

Not sure, but my plugin gets loaded inside a service or even from executable and I see no difference.

Oh... I think I missed something. I thought this was the situation:

1. Running the quickstart works
2. Running your plugin in a standalone application works
3. Running your plugin in the service fails

I think you are saying that it does not work in the second case either, I had missed that. 🤔

I downloaded the certificate from the link and have set the environment variable with its path but I don't see any differences in the result from standalone app or plugin execution. I looked into the grpc logs from my working app(executable) to see the contents of certificate exchange but don't see the data referred by new pem file. I am new to grpc and not sure if I understood your point correctly.

Hmmm... that is very odd. gRPC is pretty good at using the environment variable. Is there any way you can add some debugging code to your plugin to show if this (and any other) environment variables are set? Maybe print the output (if not null) of std::getenv("...")?

[sychopath]

I should start by saying that I am not sure what the problem is, and you should not put too much weight on what I say. I may be completely off-base and apologize if I waste your time.

Thanks @coryan for your response again. You are the best hope I have for solving this issue :)

Below is the repository of the application I am extending on. The Makefile.am will give you the gist of how the plugins get loaded dynamically once I start my unimrcp service.

https://github.com/mradmila/unimrcp

I do not recall if you mentioned this: do you use the same operating system and distribution to develop and deploy your service and the plugin?
Yes, everything is happening on the same system, ie, Ubuntu 18.04

I take it that this did not fix the problem, you just made sure that inconsistent OpenSSL versions were not the problem. Did I read that right?

I saw an older version of openssl and looked into vcpkg versions. Found vcpkg was referring to 3.0.5, so updated my system's version as well. I am not sure if this was of any use, just covering all cases.

Running the quickstart works
Yes

Running your plugin in a standalone application works

Yes, my code works fine inside a standalone application.

Running your plugin in the service fails

Exactly, it fails inside the plugin.

Hmmm... that is very odd. gRPC is pretty good at using the environment variable. Is there any way you can add some debugging code to your plugin to show if this (and any other) environment variables are set? Maybe print the output (if not null) of std::getenv("...")?

Got below logs today. Printed env variables inside the plugin and looks like it is getting the values :

2022-09-12 07:46:05:259939 [INFO] *******************Demo Recog Engine : checking env variables ----------------- : 0

******************* print env variable ------cout ------/home/ubuntu/roots.pem

******************* print env variable ------cout ------/home/ubuntu/service-account-file.json

[coryan]

Below is the repository of the application I am extending on. The Makefile.am will give you the gist of how the plugins get loaded dynamically once I start my unimrcp service.

https://github.com/mradmila/unimrcp

As best as I can tell, you are trying to write a plugin similar to this example:

https://github.com/mradmila/unimrcp/tree/master/plugins/demo-recog

it would be useful to know the exact command use to create your plugin .so. Or at least how does this .so link google-cloud-cpp and its dependencies (e.g. the -lgoogle_cloud_cpp_speech option and anything after that). It would also be useful to know if this plugin links google-cloud-cpp and its dependencies as static libraries or as shared libraries.

This is starting to remind me of:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/846

It also reminds me of:

abseil/abseil-cpp#1230

nit: your link was broken, but it is easy to cut&paste and see what you meant.

Running the quickstart works
Yes

Running your plugin in a standalone application works

Yes, my code works fine inside a standalone application.

Running your plugin in the service fails

Exactly, it fails inside the plugin.

Okay. This narrows down the problem to either (1) the environment (including file system access and environment variables) inside the plugin, (2) maybe some container issues, if your server is running in a container, (3) some weirdness with dynamically loading the code. Given your testing of the environment variables inside the plugin, I am more inclined to believe it is (3). That could be my bias, because I have had trouble with dynamic loading in the past. Seems worthwhile to rule out at least.

This is going to be difficult to troubleshoot. We need to know a few things:

• What is the output of ldd ${my-plugin-name}.so? What dependencies of the plugin are dynamically loaded?
• What are the options used to load the plugin? Probably using LD_DEBUG with the right values can give us that?
• How are symbols resolved when this library is loaded? That matters for complicated reasons explained here: https://akkadia.org/drepper/dsohowto.pdf

[sychopath]

Thanks for your response @coryan !!

Infact below is the working link we are using for microsoft stt and have plans to add google stt to it.

https://github.com/mradmila/unimrcp/tree/master/plugins/ms-recog

It would also be useful to know if this plugin links google-cloud-cpp and its dependencies as static libraries or as shared libraries.

I am attaching my Makfile.am which I have used to build by cpp code.

Make-gcp-recog.txt

nit: your link was broken, but it is easy to cut&paste and see what you meant.

my bad.

Okay. This narrows down the problem to either (1) the environment (including file system access and environment variables) inside the plugin

I was able to print the environment variables value from code, not sure if this can still be a problem.

(2) maybe some container issues, if your server is running in a container

No, simple service or executable

(3) some weirdness with dynamically loading the code. Given your testing of the environment variables inside the plugin, I am more inclined to believe it is

very high probability

What is the output of ldd ${my-plugin-name}.so? What dependencies of the plugin are dynamically loaded?

ldd gsrecog.so.0
linux-vdso.so.1 (0x00007ffdc3f30000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff8e9d43000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff8e9b24000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff8e979b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff8e93fd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff8e900c000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff8eb151000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff8e8df4000)

[coryan]

TL;DR; We need confirm:

• The Makefile.am is linking the plugin as a C++ shared object.
• What flags is unimrcp using to dlopen() the plugin, then we can use the same flags in a repro.
• We need to change the repro to link the system's OpenSSL library, this may be causing the wrong symbols to be resolved when loading the plugin.

---

Let me summarize the situation:

• Case 1 A C++ program statically linked against libgoogle_cloud_cpp_speech works.
• Case 2 A C++ program dynamically linked against libgoogle_cloud_cpp_speech (i.e. one where ldd program includes libgoogle_cloud_cpp_speech) works.
• Case 3 A C program that dynamically loads (via dlopen()) a .so which includes libgoogle_cloud_cpp_speech works.
• Case 4 Creating a plugin with the unimrcp build infrastructure and loading it into a unimrcp server does not work.

FWIW, when you said:

I am able to run sample codes using as a standalone application by building a shared object(.so) file have google stt code and calling same functions through an executable.

I thought you were referring to "Case 2". Regardless, since "Case 3" works, I think we need to concentrate on figuring out what are the differences between "Case 4" and "Case 3". Does that make sense?

I have made changes to my Makefile.am, as per your suggestions. Thanks for pointing them out. improved_Makefile.txt

I am not sure, but I think you are missing one more change. As I noted above, this may be linking that .so as a plan C library, you may need to tell libtool to link it as a C++ program. I have not used automake and libtool in a long time, and I do not recall the details of how to make the change I am proposing. Maybe this link can help you:

https://www.gnu.org/software/libtool/manual/html_node/Tags.html

To troubleshoot this further, I would need the exact command that creates the .so file. I am not sure how to do that, you may need to learn more about how automake and libtool debugging works.

I need some help here from you again. I think the flag we are referring to is "rtld_global" Can you please confirm this if I am correct ?

Yes, I am interested in the RTLD_GLOBAL flag vs. whatever else the application uses.

LD_DEBUG =all produces a huge log, but this is only where I see some data related to RTLD flags.

I expected LD_DEBUG to print the relevant info. Evidently I was wrong. This seems to work for me:

ltrace -o ltrace.log main
grep dlopen ltrace.log

Can you use something similar to find the flags for dlopen()?

With that said. I just noticed this in your output:

binding file /usr/local/unimrcp/plugin/gsrecog.so [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 [0]: normal symbol `X509_get_version'

That means the symbol resolution for an OpenSSL symbol used the symbols linked against the main program, not the symbols in gsrecog.so. That seems wrong to me, as you are probably using different OpenSSL versions in the plugin vs. the main program.

---

plugin = dlopen("/usr/src/repro-9771/plugin.so", RTLD_GLOBAL);
I see an error at my end.
Can you please confirm this behaviour ?

Confirmed.

[coryan]

I think the two versions of OpenSSL are the problem. I changed main in my repro to use the system's OpenSSL library and got a crash. I believe that using two versions and having the plugin resolve the symbols to either version is bound to produce problems. Better to use a consistent version.

I have modified my repro to have two Dockerfiles. vcpkg.Dockerfile results in crashes, local.Dockerfile works correctly.

repro-9771.tar.gz

[sychopath]

Hey @coryan ,

I think the two versions of OpenSSL are the problem.

I see 'X509_get_version' symbol is being searched in libcrypto present in /usr/lib while it should have used gsrecog.so to find that. I found a reference to openssl and crypto inside my libunimrcpserver.la file. I think while setting things up, I might have downloaded these files. I tried uninstalling openssl, but these files still exists.

https://github.com/unispeech/unimrcp/tree/master/platforms/libunimrcp-server

I have modified my repro to have two Dockerfiles. vcpkg.Dockerfile results in crashes, local.Dockerfile works correctly.

Thanks a lot for sharing this. I see an env variable declaration pointing vcpkg in your local.Dockerfile, but if I understand correctly, you do not intend to use vcpkg to build. You are building each component separately (re2, grpc, abseil,...).
Please correct me if I have mis-understood.

Not sure if it will make a a difference but at this point I would also like your suggestion if I should proceed using vcpkg or should I install each component ?

[coryan]

Hey @coryan ,

I think the two versions of OpenSSL are the problem.

I see 'X509_get_version' symbol is being searched in libcrypto present in /usr/lib while it should have used gsrecog.so to find that. I found a reference to openssl and crypto inside my libunimrcpserver.la file. I think while setting things up, I might have downloaded these files. I tried uninstalling openssl, but these files still exists.

I am not sure if there is a question there. I suspect unimrcp always needs to link libcrypto.so, so we better make

I have modified my repro to have two Dockerfiles. vcpkg.Dockerfile results in crashes, local.Dockerfile works correctly.

Thanks a lot for sharing this. I see an env variable declaration pointing vcpkg in your local.Dockerfile, but if I understand correctly, you do not intend to use vcpkg to build. You are building each component separately (re2, grpc, abseil,...). Please correct me if I have mis-understood.

That was a mistake, but had no effect. You are correct, the intent was to build each component. I have updated the repro:

repro-9771.tar.gz

Not sure if it will make a a difference but at this point I would also like your suggestion if I should proceed using vcpkg or should I install each component ?

I am going to be a bit less directive, but hopefully more general: you should make sure that gsrecog.so links against the same versions of any shared libraries used by your server. So start by listing what shared objects are used in the server (e.g. ldd my-server), then set up your build environment to use the same version of any libraries you find there. I think it would be easier to achieve this by installing each component, but you may find a different / better solution.

BTW, I don't think we have proven that the OpenSSL version problems are exactly what is causing the problems you described at the beginning of this discussion. I think it is certainly a problem, and I suspect it is likely to be the problem, but I won't be surprised if there are other issues lurking here.

[sychopath]

Hey @coryan ,

I have built each component individually and everything works smoothly. There could be multiple problems with the approach I was taking while building with vcpkg.

Thanks for helping me here !!