Cannot compile PyTorch/XLA master.

[ysiraichi]

🐛 Bug

I'm trying to compile PyTorch/XLA master branch (see command below), however I'm getting the following error:

export CUDA_HOME="/usr/local/cuda"
export XLA_CUDA=1
python setup.py develop --user

ERROR: xla/service/gpu/kernels/BUILD:156:13: Compiling xla/service/gpu/kernels/topk_kernel_bfloat16.cu.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/xla/xla/service/gpu/kernels/_objs/topk_kernel_cuda/topk_kernel_bfloat16.cu.pic.d ... (remaining 110 arguments skipped)

external/com_google_absl/absl/strings/internal/str_format/bind.h: In constructor ‘absl::lts_20230802::str_format_internal::FormatSpecTemplate<Args>::FormatSpecTemplate(const absl::lts_20230802::str_format_internal::ExtendedParsedFormat<absl::lts_20230802::FormatConversionCharSet(C)...>&)’:
external/com_google_absl/absl/strings/internal/str_format/bind.h:172:1: error: parse error in template argument list
  172 |     CheckArity<sizeof...(C), sizeof...(Args)>();
      | ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_absl/absl/strings/internal/str_format/bind.h:172:63: error: expected ‘;’ before ‘)’ token
  172 |     CheckArity<sizeof...(C), sizeof...(Args)>();
      |                                                               ^
      |                                                               ;
external/com_google_absl/absl/strings/internal/str_format/bind.h:173:147: error: template argument 1 is invalid
  173 |     CheckMatches<C...>(absl::make_index_sequence<sizeof...(C)>{});
      |                                                                                                                                                   ^
external/com_google_absl/absl/strings/internal/str_format/bind.h:173:151: error: expected primary-expression before ‘{’ token
  173 |     CheckMatches<C...>(absl::make_index_sequence<sizeof...(C)>{});
      |                                                                                                                                                       ^
external/com_google_absl/absl/strings/internal/str_format/bind.h:173:150: error: expected ‘;’ before ‘{’ token
  173 |     CheckMatches<C...>(absl::make_index_sequence<sizeof...(C)>{});
      |                                                                                                                                                      ^
      |                                                                                                                                                      ;
external/com_google_absl/absl/strings/internal/str_format/bind.h:173:153: error: expected primary-expression before ‘)’ token
  173 |     CheckMatches<C...>(absl::make_index_sequence<sizeof...(C)>{});
      |                                                                                                                                                         ^
Target //:_XLAC.so failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 882.296s, Critical Path: 94.86s
INFO: 8108 processes: 259 internal, 7849 local.
FAILED: Build did NOT complete successfully
error: command 'bazel' failed with exit status 1

Environment

• PyTorch version: becfda005e524f93b1efed64917a129ef6778135
• PyTorch/XLA version: f4971a7

cc @miladm @lezcano

[ysiraichi]

After investigating a bit, I see that the last XLA pin update is the culprit (#6530). Replacing the updated XLA version b166243711f71b0a55daa1eda36b1dc745886784 by the former pin c08cfb0377e4e33a21bde65950f986a21c8a8199 made the error go away.

[yeounoh]

After investigating a bit, I see that the last XLA pin update is the culprit (#6530). Replacing the updated XLA version b166243711f71b0a55daa1eda36b1dc745886784 by the former pin c08cfb0377e4e33a21bde65950f986a21c8a8199 made the error go away.

Hi @ysiraichi Does this mean we have to revert the latest pin update? Is it resolved in the HEAD? We are moving to the HEAD soon.