OTLP example fails to build with error "undefined reference to `grpc::Status::OK' "

[VivekSubr]

Describe your environment
WSL ubuntu jammy

Steps to reproduce
Install grpc as per doc: https://grpc.io/docs/languages/cpp/quickstart/
Build opentelemetry,

cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=ON -DWITH_ABSEIL=OFF -DWITH_OTLP=ON -DBUILD_TESTING=OFF
cmake --build . --target all

What is the expected behavior?
Build should go through

What is the actual behavior?
Got error,

[ 79%] Linking CXX executable example_otlp_grpc
cd /home/vivek/Client-Server/module/opentelemetry-cpp/build/examples/otlp && /usr/bin/cmake -E cmake_link_script CMakeFiles/example_otlp_grpc.dir/link.txt --verbose=1
/usr/bin/c++ -rdynamic CMakeFiles/example_otlp_grpc.dir/grpc_main.cc.o -o example_otlp_grpc  -Wl,-rpath,/home/vivek/Client-Server/module/opentelemetry-cpp/build/examples/common/foo_library:/home/vivek/Client-Server/module/opentelemetry-cpp/build/exporters/otlp:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/trace:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/resource:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/metrics:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/common ../common/foo_library/libcommon_foo_library.so ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc.so ../../exporters/otlp/libopentelemetry_otlp_recordable.so ../../sdk/src/trace/libopentelemetry_trace.so ../../sdk/src/resource/libopentelemetry_resources.so ../../sdk/src/metrics/libopentelemetry_metrics.so ../../sdk/src/common/libopentelemetry_common.so ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc_client.so ../../libopentelemetry_proto.a /usr/lib/x86_64-linux-gnu/libprotobuf.a
/usr/bin/ld: ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc.so: undefined reference to `grpc::Status::OK'
collect2: error: ld returned 1 exit status

Additional context
grpc installation looks fine,

vivek@LAPTOP-7E1991MU:/usr/lib$ ls | grep "grpc"
libgrpc++.a
libgrpc++_alts.a
libgrpc++_error_details.a
libgrpc++_reflection.a
libgrpc++_unsecure.a
libgrpc.a
libgrpc_authorization_provider.a
libgrpc_plugin_support.a
libgrpc_unsecure.a
libgrpcpp_channelz.a

I did the exact steps provided by grpc project, but looks like this build can't link to libgrpc

[lalitb]

I may be wrong, but looks similar to #1940 though it is with Alpine which brings gRPC package.

/usr/lib/gcc/aarch64-alpine-linux-musl/11.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: CMakeFiles/otlp_grpc_exporter_test.dir/test/otlp_grpc_exporter_test.cc.o: undefined reference to symbol '_ZN4grpc6Status2OKE'

Is it with gRPC 1.52.1 ?

[VivekSubr]

@lalitb - yes, 1.52,
git clone --recurse-submodules -b v1.52.0 --depth 1 --shallow-submodules https://github.com/grpc/grpc

Was it working with an older version?

I tried the solution mentioned there, adding -DWITH_STL=ON and the LDFLAGS,

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build$ echo $LDFLAGS
-Wl,--copy-dt-needed-entries

Didn't work for me.

[owent]

Could you please post the logs of linking error after export LDFLAGS=-Wl,--copy-dt-needed-entries ? Just like [ 79%] Linking CXX executable example_otlp_grpc ... above.

grpc::Status::OK shoud be in libgrpc++.a and it should be linked into libopentelemetry_exporter_otlp_grpc_client.so.Could you please also run these commands and post the output?

nm /usr/lib/libgrpc++.a | grep Status | grep OK | c++filt
nm /home/vivek/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/libopentelemetry_exporter_otlp_grpc_client.so  | grep Status | grep OK | c++filt
readelf -d /home/vivek/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/libopentelemetry_exporter_otlp_grpc.so | grep NEEDED

[VivekSubr]

@owent - no change in error with the LDFLAGS,

[ 79%] Linking CXX executable example_otlp_grpc
cd /home/vivek/Client-Server/module/opentelemetry-cpp/build/examples/otlp && /usr/bin/cmake -E cmake_link_script CMakeFiles/example_otlp_grpc.dir/link.txt --verbose=1
/usr/bin/c++ -Wl,--copy-dt-needed-entries -rdynamic CMakeFiles/example_otlp_grpc.dir/grpc_main.cc.o -o example_otlp_grpc  -Wl,-rpath,/home/vivek/Client-Server/module/opentelemetry-cpp/build/examples/common/foo_library:/home/vivek/Client-Server/module/opentelemetry-cpp/build/exporters/otlp:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/trace:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/resource:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/metrics:/home/vivek/Client-Server/module/opentelemetry-cpp/build/sdk/src/common ../common/foo_library/libcommon_foo_library.so ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc.so ../../exporters/otlp/libopentelemetry_otlp_recordable.so ../../sdk/src/trace/libopentelemetry_trace.so ../../sdk/src/resource/libopentelemetry_resources.so ../../sdk/src/metrics/libopentelemetry_metrics.so ../../sdk/src/common/libopentelemetry_common.so ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc_client.so ../../libopentelemetry_proto.a /usr/lib/x86_64-linux-gnu/libprotobuf.a
/usr/bin/ld: ../../exporters/otlp/libopentelemetry_exporter_otlp_grpc.so: undefined reference to `grpc::Status::OK'
collect2: error: ld returned 1 exit status

CMake command used was,
cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=ON -DWITH_ABSEIL=OFF -DWITH_STL=ON -DWITH_OTLP=ON -DBUILD_TESTING=OFF

As for the commands,

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build$ nm /usr/lib/libgrpc++.a | grep Status | grep OK | c++filt
                 U grpc::Status::OK
                 U grpc::Status::OK
                 U grpc::Status::OK
0000000000000000 B _ZGRN4grpc6Status2OKE_
0000000000000048 B grpc::Status::OK

Output for the second command is empty,

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build$ nm /home/vivek/Client-Server/module/opentelemetry-cpp/build/exp
orters/otlp/libopentelemetry_exporter_otlp_grpc_client.so  | grep Status | grep OK | c++filt
vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build$

And,

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build$ readelf -d /home/vivek/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/libopentelemetry_exporter_otlp_grpc.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libopentelemetry_otlp_recordable.so]
 0x0000000000000001 (NEEDED)             Shared library: [libopentelemetry_exporter_otlp_grpc_client.so]
 0x0000000000000001 (NEEDED)             Shared library: [libopentelemetry_common.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

Don't understand any of this stuff, sorry. :)

[owent]

The problem seems to be libopentelemetry_exporter_otlp_grpc_client.so do not link grpc++ or the symbols is stripped. Could you please also provide the linking flags of libopentelemetry_exporter_otlp_grpc_client.so?
If you are using ninja, it may be build exporters/otlp/libopentelemetry_exporter_otlp_grpc_client.so section in ~/Client-Server/module/opentelemetry-cpp/build/build.ninja .
Or if you are using Makefile, the flags may be found in flags.make and link.txt in ~/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir/.
Thanks for your patient.

[lalitb]

The problem seems to be libopentelemetry_exporter_otlp_grpc_client.so do not link grpc++ or the symbols is stripped.

In Alpine case (i.e. #1940), There were other symbols (so not stripped) but the gRPC++ symbols were missing. I will spend some time tomorrow to debug #1940 further.

[VivekSubr]

@owent, this right?

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir$ cat link.txt
/usr/bin/c++ -fPIC -Wl,--copy-dt-needed-entries -shared -Wl,-soname,libopentelemetry_exporter_otlp_grpc_client.so -o libopentelemetry_exporter_otlp_grpc_client.so CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir/src/otlp_grpc_client.cc.o CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir/src/otlp_grpc_utils.cc.o  ../../libopentelemetry_proto.a /usr/lib/libgrpc++.a /usr/lib/x86_64-linux-gnu/libprotobuf.a /usr/lib/libgrpc.a /usr/lib/libcares.a /usr/lib/libaddress_sorting.a /usr/lib/libre2.a /usr/lib/libupb.a /usr/lib/x86_64-linux-gnu/libabsl_raw_hash_set.a /usr/lib/x86_64-linux-gnu/libabsl_hashtablez_sampler.a /usr/lib/x86_64-linux-gnu/libabsl_hash.a /usr/lib/x86_64-linux-gnu/libabsl_city.a /usr/lib/x86_64-linux-gnu/libabsl_low_level_hash.a /usr/lib/x86_64-linux-gnu/libabsl_statusor.a /usr/lib/libgpr.a /usr/lib/libz.a -ldl -lm -lrt /usr/lib/x86_64-linux-gnu/libabsl_bad_variant_access.a /usr/lib/x86_64-linux-gnu/libabsl_status.a /usr/lib/x86_64-linux-gnu/libabsl_strerror.a /usr/lib/x86_64-linux-gnu/libabsl_random_distributions.a /usr/lib/x86_64-linux-gnu/libabsl_random_seed_sequences.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_pool_urbg.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_hwaes.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_hwaes_impl.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_slow.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_platform.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_seed_material.a /usr/lib/x86_64-linux-gnu/libabsl_random_seed_gen_exception.a /usr/lib/x86_64-linux-gnu/libabsl_cord.a /usr/lib/x86_64-linux-gnu/libabsl_bad_optional_access.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_info.a /usr/lib/x86_64-linux-gnu/libabsl_cord_internal.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_functions.a /usr/lib/x86_64-linux-gnu/libabsl_exponential_biased.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_handle.a /usr/lib/x86_64-linux-gnu/libabsl_str_format_internal.a /usr/lib/x86_64-linux-gnu/libabsl_synchronization.a /usr/lib/x86_64-linux-gnu/libabsl_stacktrace.a /usr/lib/x86_64-linux-gnu/libabsl_symbolize.a /usr/lib/x86_64-linux-gnu/libabsl_debugging_internal.a /usr/lib/x86_64-linux-gnu/libabsl_demangle_internal.a /usr/lib/x86_64-linux-gnu/libabsl_graphcycles_internal.a /usr/lib/x86_64-linux-gnu/libabsl_malloc_internal.a /usr/lib/x86_64-linux-gnu/libabsl_time.a /usr/lib/x86_64-linux-gnu/libabsl_strings.a /usr/lib/x86_64-linux-gnu/libabsl_throw_delegate.a /usr/lib/x86_64-linux-gnu/libabsl_int128.a /usr/lib/x86_64-linux-gnu/libabsl_strings_internal.a /usr/lib/x86_64-linux-gnu/libabsl_base.a /usr/lib/x86_64-linux-gnu/libabsl_spinlock_wait.a -lrt /usr/lib/x86_64-linux-gnu/libabsl_raw_logging_internal.a /usr/lib/x86_64-linux-gnu/libabsl_log_severity.a /usr/lib/x86_64-linux-gnu/libabsl_civil_time.a /usr/lib/x86_64-linux-gnu/libabsl_time_zone.a /usr/lib/libssl.a /usr/lib/libcrypto.a -lpthread

It has /usr/lib/libgrpc.a linked, not /usr/lib/libgrpc++.a as you said... and libgrpc.a doesn't have the symbol.

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir$ nm /usr/lib/libgrpc.a | grep Status | grep OK | c++filt

[owent]

@owent, this right?

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir$ cat link.txt
/usr/bin/c++ -fPIC -Wl,--copy-dt-needed-entries -shared -Wl,-soname,libopentelemetry_exporter_otlp_grpc_client.so -o libopentelemetry_exporter_otlp_grpc_client.so CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir/src/otlp_grpc_client.cc.o CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir/src/otlp_grpc_utils.cc.o  ../../libopentelemetry_proto.a /usr/lib/libgrpc++.a /usr/lib/x86_64-linux-gnu/libprotobuf.a /usr/lib/libgrpc.a /usr/lib/libcares.a /usr/lib/libaddress_sorting.a /usr/lib/libre2.a /usr/lib/libupb.a /usr/lib/x86_64-linux-gnu/libabsl_raw_hash_set.a /usr/lib/x86_64-linux-gnu/libabsl_hashtablez_sampler.a /usr/lib/x86_64-linux-gnu/libabsl_hash.a /usr/lib/x86_64-linux-gnu/libabsl_city.a /usr/lib/x86_64-linux-gnu/libabsl_low_level_hash.a /usr/lib/x86_64-linux-gnu/libabsl_statusor.a /usr/lib/libgpr.a /usr/lib/libz.a -ldl -lm -lrt /usr/lib/x86_64-linux-gnu/libabsl_bad_variant_access.a /usr/lib/x86_64-linux-gnu/libabsl_status.a /usr/lib/x86_64-linux-gnu/libabsl_strerror.a /usr/lib/x86_64-linux-gnu/libabsl_random_distributions.a /usr/lib/x86_64-linux-gnu/libabsl_random_seed_sequences.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_pool_urbg.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_hwaes.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_hwaes_impl.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_randen_slow.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_platform.a /usr/lib/x86_64-linux-gnu/libabsl_random_internal_seed_material.a /usr/lib/x86_64-linux-gnu/libabsl_random_seed_gen_exception.a /usr/lib/x86_64-linux-gnu/libabsl_cord.a /usr/lib/x86_64-linux-gnu/libabsl_bad_optional_access.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_info.a /usr/lib/x86_64-linux-gnu/libabsl_cord_internal.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_functions.a /usr/lib/x86_64-linux-gnu/libabsl_exponential_biased.a /usr/lib/x86_64-linux-gnu/libabsl_cordz_handle.a /usr/lib/x86_64-linux-gnu/libabsl_str_format_internal.a /usr/lib/x86_64-linux-gnu/libabsl_synchronization.a /usr/lib/x86_64-linux-gnu/libabsl_stacktrace.a /usr/lib/x86_64-linux-gnu/libabsl_symbolize.a /usr/lib/x86_64-linux-gnu/libabsl_debugging_internal.a /usr/lib/x86_64-linux-gnu/libabsl_demangle_internal.a /usr/lib/x86_64-linux-gnu/libabsl_graphcycles_internal.a /usr/lib/x86_64-linux-gnu/libabsl_malloc_internal.a /usr/lib/x86_64-linux-gnu/libabsl_time.a /usr/lib/x86_64-linux-gnu/libabsl_strings.a /usr/lib/x86_64-linux-gnu/libabsl_throw_delegate.a /usr/lib/x86_64-linux-gnu/libabsl_int128.a /usr/lib/x86_64-linux-gnu/libabsl_strings_internal.a /usr/lib/x86_64-linux-gnu/libabsl_base.a /usr/lib/x86_64-linux-gnu/libabsl_spinlock_wait.a -lrt /usr/lib/x86_64-linux-gnu/libabsl_raw_logging_internal.a /usr/lib/x86_64-linux-gnu/libabsl_log_severity.a /usr/lib/x86_64-linux-gnu/libabsl_civil_time.a /usr/lib/x86_64-linux-gnu/libabsl_time_zone.a /usr/lib/libssl.a /usr/lib/libcrypto.a -lpthread

It has /usr/lib/libgrpc.a linked, not /usr/lib/libgrpc++.a as you said... and libgrpc.a doesn't have the symbol.

vivek@LAPTOP-7E1991MU:~/Client-Server/module/opentelemetry-cpp/build/exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_client.dir$ nm /usr/lib/libgrpc.a | grep Status | grep OK | c++filt

Thanks for your information, I have raised a PR(#2005 ) to fix this problem. The problem is opentelemetry_exporter_otlp_grpc_client do not use some APIs of gRPC, and some symbols are stripped when linking.
This PR let opentelemetry_exporter_otlp_grpc_client delegate these calls.