undefined reference to '__cudaRegisterLinkedBinary' when adding a CUDA library

[grueyg]

Hello,

I am attempting to add a CUDA library to my project, but I encountered many errors like undefined reference to '__cudaRegisterLinkedBinary' during compilation,

The library itself seems to be functioning correctly, and I suspect the problem might be related to how I’ve integrated the library into my Bazel build setup. Below are the details of my setup.

test_phantom.cc:

#include <iostream>
#include "phantom.h"

using namespace std;
using namespace phantom;

void run_tests() {
    EncryptionParameters parms(scheme_type::bgv);
    cout << "Running tests..." << endl;
}

int main() {
    run_tests();
    return 0;
}

BUILD file for test_phantom:

cc_binary(
    name = "test_phantom",
    srcs = ["test_phantom.cc"],
    copts = [
        "-std=c++17",
    ],
    deps = [
        "//third_party/phantom/install:phantom",
        "@local_cuda//:cuda_runtime",
    ],
)

BUILD file for CUDA library:

cc_library(
    name = "phantom",
    hdrs = glob([
        "include/phantom/*.h",
        "include/phantom/*.cuh",
    ]),
    srcs = ["lib/libPhantom.a"],
    linkstatic = True,
    includes = [
        "include/phantom",
    ],
    deps = [
        "@local_cuda//:cuda_runtime",
    ]
)

When I run the following command:

bazel build //tests:test_phantom --@rules_cuda//cuda:enable=True

I receive the following errors:

INFO: Analyzed target //tests:test_phantom (1 packages loaded, 2 targets configured).
ERROR: /home/user/project/sf/bazel-cuda-test/tests/BUILD.bazel:24:10: Linking tests/test_phantom failed: (Exit 1): gcc failed: error executing CppLink command (from target //tests:test_phantom) /usr/bin/gcc @bazel-out/k8-fastbuild/bin/tests/test_phantom-2.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
third_party/phantom/install/lib/libPhantom.a(modulus.cu.o):tmpxft_00007bf6_00000000-6_modulus.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_f2422dbf_10_modulus_cu_05592024'
third_party/phantom/install/lib/libPhantom.a(numth.cu.o):tmpxft_00007c03_00000000-6_numth.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_29298749_8_numth_cu_13f6fc3a'
third_party/phantom/install/lib/libPhantom.a(uintarith.cu.o):tmpxft_00007c0c_00000000-6_uintarith.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_d2b8fd3b_12_uintarith_cu_9dca457a'
third_party/phantom/install/lib/libPhantom.a(uintarithsmallmod.cu.o):tmpxft_00007c21_00000000-6_uintarithsmallmod.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_6d3d1c83_20_uintarithsmallmod_cu_aea87feb'
third_party/phantom/install/lib/libPhantom.a(globals.cu.o):tmpxft_00007bee_00000000-6_globals.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_f290d988_10_globals_cu_c6bcbb62'
collect2: error: ld returned 1 exit status
Target //tests:test_phantom failed to build
INFO: Analyzed target //tests:test_phantom (1 packages loaded, 2 targets configured).
ERROR: /home/user/project/sf/bazel-cuda-test/tests/BUILD.bazel:24:10: Linking tests/test_phantom failed: (Exit 1): gcc failed: error executing CppLink command (from target //tests:test_phantom) /usr/bin/gcc @bazel-out/k8-fastbuild/bin/tests/test_phantom-2.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
third_party/phantom/install/lib/libPhantom.a(modulus.cu.o):tmpxft_00007bf6_00000000-6_modulus.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_f2422dbf_10_modulus_cu_05592024'
third_party/phantom/install/lib/libPhantom.a(numth.cu.o):tmpxft_00007c03_00000000-6_numth.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_29298749_8_numth_cu_13f6fc3a'
third_party/phantom/install/lib/libPhantom.a(uintarith.cu.o):tmpxft_00007c0c_00000000-6_uintarith.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_d2b8fd3b_12_uintarith_cu_9dca457a'
third_party/phantom/install/lib/libPhantom.a(uintarithsmallmod.cu.o):tmpxft_00007c21_00000000-6_uintarithsmallmod.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_6d3d1c83_20_uintarithsmallmod_cu_aea87feb'
third_party/phantom/install/lib/libPhantom.a(globals.cu.o):tmpxft_00007bee_00000000-6_globals.cudafe1.cpp:function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary_f290d988_10_globals_cu_c6bcbb62'
collect2: error: ld returned 1 exit status
Target //tests:test_phantom failed to build

I looked into the relevant information about this error, and it seems to be caused by 'relocatable device code linking.' However, most of the available information is about CMake compilation, and there is a lack of solutions for resolving this error in Bazel. I tried adding the following to test_phantom...

Let me know if you need any further adjustments!

I also tried adding the following linkopts to the test_phantom rule:

linkopts = [
    "-lcudart",
    "-lcudadevrt",
],

But this did not resolve the issue, and the same error persists.

I’m not sure if this is an issue with how I am integrating the CUDA library, or if there's something else I am missing in my Bazel setup. Could you help me identify the root cause of this error? Any guidance or suggestions would be greatly appreciated.

Thank you very much for your help!

[grueyg]

I can successfully compile test_phantom.cc using the command:

nvcc -o test_phantom tests/test_phantom.cc \
-Ithird_party/phantom/install/include/phantom \
-Lthird_party/phantom/install/lib \
-lPhantom \
-L/usr/local/cuda/lib64 \
-lcudart -std=c++17

However, in Bazel, even though I set test_phantom as a cuda_binary, I still get the same error. Here’s my Bazel configuration:

cuda_binary(
    name = "test_phantom",
    srcs = ["test_phantom.cc"],
    copts = [
        "-std=c++17",
    ],
    linkopts = [
        "-L/usr/local/cuda/lib64",
        "-lcudart",
    ],
    deps = [
        "//third_party/phantom/install:phantom",
        "@local_cuda//:cuda_runtime",
    ],
)

[cloudhan]

First you don't use these type of linkopts in your cuda_binary

    linkopts = [
        "-L/usr/local/cuda/lib64",
        "-lcudart",
    ],

They are implicit, otherwise, there will be a bug.

Second, it seems you are relying on the relocateable device code, try set rdc=True
See examples/rdc/BUILD.bazel

[grueyg]

@cloudhan ，thank you for your reply and help!

Following your guidance, I tried the configuration below, but I still received a similar error: function __sti____cudaRegisterAll(): error: undefined reference to '__cudaRegisterLinkedBinary'.

cuda_objects(
    name = "test_object",
    srcs = ["test_phantom.cc"],
    copts = [
        "-std=c++17",
    ],
    deps = [
        "//third_party/phantom/install:phantom",
        "@local_cuda//:cuda_runtime",
    ],
)

cuda_library(
    name = "test_lib",
    rdc = True,
    deps = [
        ":test_object"
    ],
)

cuda_binary(
    name = "test_phantom",
    deps = [
        ":test_lib",
    ],
)

Additionally, I'm not sure if this is related to relocatable device code, because I found that the following command also compiles successfully (without -lcudart):

nvcc -o test_phantom tests/test_phantom.cc -Ithird_party/phantom/install/include/phantom -Lthird_party/phantom/install/lib -lPhantom -std=c++17

[cloudhan]

It seem you are working (relying) on some OSS project, it will be more approachable if you provide me a minimal reproduce repo. This might be a rdc or whole archive linking or maybe combined in your phantom lib.

I consider it being related to rdc, pure judged from the __cudaRegisterLinkedBinary_blahblah multiple definitions or missing definition only happened with rdc related compilation.

[grueyg]

Hi, @cloudhan . Although I don't know the cause of this error, it has been resolved. Previously, I compiled the CUDA library into a static library using CMake and linked it to my project, which caused the aforementioned error. Following your guidance, I tried to create a minimal reproducible example. During this process, I built the corresponding cuda_library for each module of the CUDA library and compiled it using Bazel. Surprisingly, this approach worked very well.

Once again, I'm really grateful for your help and guidance. This project is really great!