Issue with cxx_include_drectories in default c++ toolchain.

[gunan]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

bazel builds on redhat and centos fail with:
undeclared inclusion(s) in rule ...
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stddef.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdarg.h'

More information on:
tensorflow/tensorflow#14380

Looks like default toolchain is having problems locating default C++ headers.

If asking a question or requesting a feature, also tell us about the underlying problem you're trying to solve.

If possible, provide a minimal example to reproduce the problem:

git clone http://github.com/tensorflow/tensorflow
cd tensorflow
yes "" | ./configure
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Environment info

• Operating System:
  redhat, centos

• Bazel version (output of bazel info release):
  seems to be any version.

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web?

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)
tensorflow/tensorflow#14380

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

[hlopko]

I'm bazel sheriff this week and I'm going for 3 weeks long vacation on Friday, so I cannot promise I'll have this investigated and fixed :(

The error is a result of dependency file validation. C compiler gave us a .d file with headers that were not reported in cxx_builtin_include_directory fields in the CROSSTOOL and is complaining that the target didn't declare all its inputs.

In autoconfigurated CROSSTOOL, this is the line that is producing those fields: https://source.bazel.build/bazel/+/master:tools/cpp/unix_cc_configure.bzl;l=151?q=unix_cc_configure
I assume that C compiler is not reporting all its include directories correctly.

Summoning @ilya-biryukov since he saved my back in these situatiosn many times before :)

[gunan]

A user with CentOS just reported this output from the command in the unix_cc_configure:

$ gcc -E  -xc++ - -v
.....
#include <...> search starts here:
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include
/usr/local/include
/usr/include
End of search list.

The headers that are complained about are in:

'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stddef.h'
'/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdarg.h'

Which should be covered by the 4th folder in the output.
Maybe these are issues with our GPU crosstool. I will check if I can build tf on centos using docker.

[gunan]

It is possible our GPU crosstool is the culprit here. I tried to reproduce this on CPU builds, and things seem to work OK for me for CPU builds.
Is there an easy way for me to reuse get_escaped_cxx_inc_directories rule in our GPU crosstool, or the bzl files around it? Or do you recommend I just copy the routine in our bzl files?

[gunan]

Digging deeper, I was not able to reproduce the problem with the latest bazel versions. I will close this issue for now.

[ilya-biryukov]

Tensorflow does not use bazel's autoconfigure to produce the crosstool, using this script instead:
https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl
I've hit similar issues before, will try to follow-up on the tensorflow issue.