Toolchain resolution seemingly failing and falling back to /usr/bin/gcc in some cases

[bsirang]

I'm having trouble with the bazel-embedded rules in some cases. In a plain Ubuntu host, I can build my target with --platforms=@bazel_embedded//platforms:cortex_m7_fpu and it works fine and invokes external/bazel_embedded/toolchains/gcc_arm_none_eabi/gcc_wrappers/nix/gcc

(cd /home/parallels/.cache/bazel/_bazel_parallels/14af9fa9524e44cf06682e101a7eaa03/execroot/haystack && \
  exec env - \
    PATH=/home/parallels/.cache/bazelisk/downloads/bazelbuild/bazel-4.2.1-linux-x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/parallels/go/bin:/home/parallels/gcc/gcc-arm-none-eabi-10-2020-q4-major/bin:/home/parallels/.gem/ruby/2.7.0/bin \
    PWD=/proc/self/cwd \
  external/bazel_embedded/toolchains/gcc_arm_none_eabi/gcc_wrappers/nix/gcc

However when I run the same thing in a docker environment (based on Ubuntu) it attempts to use /usr/bin/gcc and fails because -mthumb doesn't exist.

(cd /root/.cache/bazel/_bazel_root/18e8352205c71478753c3b63796033c0/execroot/haystack && \
  exec env - \
    PATH=/root/.cache/bazelisk/downloads/bazelbuild/bazel-4.2.1-linux-x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc ...

I used the --toolchain_resolution_debug flag and interestingly, in both the working and non-working case it output the exact same thing (I diff'd and they were exactly the same):

Starting local Bazel server and connecting to it...
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv6-m_none; mismatching values: armv6-m, none
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7-m_none; mismatching values: armv7-m, none
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv4-sp-d16; mismatching values: fpv4-sp-d16
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: execution @local_config_platform//:host: Selected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m0: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m1: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m3: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m4: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m4_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m7: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32f2xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32f3xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32f4xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32f7xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32g4xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @bazel_embedded//platforms:stm32h7xx: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_none; mismatching values: none
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm, android
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @bazel_embedded//platforms:cortex_m7_fpu: Rejected toolchain @local_config_cc//:cc-compiler-k8; mismatching values: x86_64, linux
INFO: ToolchainResolution: Target platform @bazel_embedded//platforms:cortex_m7_fpu: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @bazel_embedded//toolchains/gcc_arm_none_eabi:arm_none_eabi_armv7e-m_fpv5-d16
INFO: ToolchainResolution: Target platform @bazel_embedded//platforms:cortex_m7_fpu: Selected execution platform @local_config_platform//:host, 
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, 
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, 

[bsirang]

More info:

Here's the bazel version I'm using: bazel 4.2.1

Here's the working case I have so far:

• Regular user in Ubuntu VM

Here's the non-working cases I have found so far:

• root user (via sudo command) in Ubuntu VM
• docker environment (as root)
• docker environment (as regular user with UID 1000)

[bsirang]

Perhaps this is relevant: bazelbuild/bazel#12712

[nathaniel-brough]

I have run into this once or twice, but haven't really been able to zero in on it and reproduce it once it starts working again.

Here's the working case I have so far:
Regular user in Ubuntu VM

What version of ubuntu are you running?

Here's the non-working cases I have found so far:

• root user (via sudo command) in Ubuntu VM
• docker environment (as root)

This should work, but I would recommend against using root for you builds for security reasons.

Just as a hunch I would guess that this is something to do with how your bazelrc is being resolved with different users. Do you have a .bazelrc file in your workspace or in your home directory? The first thing I would try is to ensure that you have a .bazelrc in your workspace and that it is present in your Docker container. It should contain at least the following;

# Enable toolchain resolution with cc
build --incompatible_enable_cc_toolchain_resolution

Or alternatively, just add --incompatible_enable_cc_toolchain_resolution to your command line.

If debugging the .bazelrc doesn't work would you mind posting a minimal set of steps to reproduce the problem i.e. basic workspace + Dockerfile etc? If the project is open source a URL should suffice.

[bsirang]

Ahh everything makes sense now...

You were right... I inadvertently had a .bazelrc in my home directory in addition to my workspace. I had that --incompatible_enable_cc_toolchain_resolution flag there hence why it worked in one of the four scenarios I listed.

It's still interesting the --toolchain_resolution_debug didn't help pinpoint the issue.

Anyway, I'll go ahead and close this one out.