cc-wrapper does not pass -stdlib=libc++ as part of cxxflags

[rrbutani]

The Bug

Currently cc-wrapper does add -stdlib=libc++ to libcxx-ldflags but not to libcxx-cxxflags.

This leads to clang appending gcc libstdc++ header search directories to the search path which in turn breaks libc++ headers (#include_next <...> no longer finds libc headers, as intended, but now finds libstdc++ headers).

To Reproduce

nix-shell -p llvmPackages_14.libcxxStdenv.cc --run "echo '#include <string>' | clang++ -c -xc++ -" --pure

^ results in errors that look like this:

Click to expand.

In file included from <stdin>:1:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/string:523:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__functional_base:22:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/exception:83:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/cstdlib:85:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/stdlib.h:93:
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/stdlib.h:38:12: error: no member named 'abort' in namespace 'std'
using std::abort;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/stdlib.h:39:12: error: no member named 'atexit' in namespace 'std'
using std::atexit;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/stdlib.h:40:12: error: no member named 'exit' in namespace 'std'
using std::exit;
      ~~~~~^
In file included from <stdin>:1:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/string:523:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__functional_base:22:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/exception:83:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/cstdlib:85:
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/stdlib.h:148:34: error: unknown type name 'ldiv_t'
inline _LIBCPP_INLINE_VISIBILITY ldiv_t div(long __x, long __y) _NOEXCEPT {
                                 ^
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/stdlib.h:149:12: error: no member named 'ldiv' in the global namespace
  return ::ldiv(__x, __y);
         ~~^
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/stdlib.h:152:34: error: unknown type name 'lldiv_t'
inline _LIBCPP_INLINE_VISIBILITY lldiv_t div(long long __x,
                                 ^
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/stdlib.h:154:12: error: no member named 'lldiv' in the global namespace
  return ::lldiv(__x, __y);
         ~~^
In file included from <stdin>:1:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/string:523:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__functional_base:23:
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/new:327:11: error: no member named 'posix_memalign' in the global namespace
  (void)::posix_memalign(&__result, __alignment, __size);
        ~~^
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/new:338:5: error: no type named 'free' in the global namespace
  ::free(__ptr);
  ~~^
In file included from <stdin>:1:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/string:523:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__functional_base:26:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/utility:236:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/compare:144:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/compare_partial_order_fallback.h:13:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/partial_order.h:14:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/weak_order.h:14:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/strong_order.h:18:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/cmath:308:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/math.h:300:
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:39:12: error: no member named 'acos' in namespace 'std'
using std::acos;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:40:12: error: no member named 'asin' in namespace 'std'
using std::asin;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:41:12: error: no member named 'atan' in namespace 'std'
using std::atan;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:42:12: error: no member named 'atan2' in namespace 'std'
using std::atan2;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:43:12: error: no member named 'cos' in namespace 'std'; did you mean 'ios'?
using std::cos;
      ~~~~~^
/nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/iosfwd:161:38: note: 'ios' declared here
typedef basic_ios<char>              ios;
                                     ^
In file included from <stdin>:1:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/string:523:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__functional_base:26:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/utility:236:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/compare:144:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/compare_partial_order_fallback.h:13:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/partial_order.h:14:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/weak_order.h:14:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/__compare/strong_order.h:18:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/cmath:308:
In file included from /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1/math.h:300:
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:44:12: error: no member named 'sin' in namespace 'std'
using std::sin;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:45:12: error: no member named 'tan' in namespace 'std'
using std::tan;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:46:12: error: no member named 'cosh' in namespace 'std'
using std::cosh;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:47:12: error: no member named 'sinh' in namespace 'std'
using std::sinh;
      ~~~~~^
/nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/math.h:48:12: error: no member named 'tanh' in namespace 'std'
using std::tanh;
      ~~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

The Fix

Passing in --std=libc++ inhibits clang from adding the gcc libstdc++ header directories to the search path and fixes this error:

nix-shell -p llvmPackages_14.libcxxStdenv.cc --run "echo '#include <string>' | clang++ -c -xc++ - --stdlib=libc++" --pure

Running with -v lets us compare the list of search directories:

• clang++ -c -xc++ - -v:

  #include "..." search starts here:
  #include <...> search starts here:
   /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include
   /nix/store/azb7l7hzicncdyz1phnjwy0pjb97ikjl-libcxxabi-14.0.1-dev/include
   /nix/store/870xakmqvll2paqgn0s2x959wkxd02jm-compiler-rt-libc-14.0.1-dev/include
   /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1
   /nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0
   /nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/x86_64-unknown-linux-gnu
   /nix/store/2lr0fc0ak8rwj0k8n3shcyz1hz63wzma-gcc-11.3.0/lib64/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/backward
   /nix/store/n93g9kfx1sy6mpiy4z7hzjyf9r7qn9jz-clang-wrapper-14.0.1/resource-root/include
   /nix/store/k3d8wqlsnmm5270zd19cbs26g7wifxj6-glibc-2.34-210-dev/include
  

• clang++ -c -xc++ - --stdlib=libc++ -v:

  #include "..." search starts here:
  #include <...> search starts here:
   /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include
   /nix/store/azb7l7hzicncdyz1phnjwy0pjb97ikjl-libcxxabi-14.0.1-dev/include
   /nix/store/870xakmqvll2paqgn0s2x959wkxd02jm-compiler-rt-libc-14.0.1-dev/include
   /nix/store/rd7zvb5ic6agrsfkp06w3bahflzb35vg-libcxx-14.0.1-dev/include/c++/v1
   /nix/store/n93g9kfx1sy6mpiy4z7hzjyf9r7qn9jz-clang-wrapper-14.0.1/resource-root/include
   /nix/store/k3d8wqlsnmm5270zd19cbs26g7wifxj6-glibc-2.34-210-dev/include
  

This (i.e. adding echo "-stdlib=libc++" >> $out/nix-support/libcxx-cxxflags here) seems like an uncontroversial change to me but I thought I'd open an issue about this first just in case I'm missing something obvious or just not using the wrapper compiler correctly.

Notify maintainers

CODEOWNERS for cc-wrapper points to @Ericson2314; apologies if this is incorrect.

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.63, NixOS, 22.11 (Raccoon), 22.11.20220901.2da64a8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.0`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[wegank]

@rrbutani Could you please file the PR? Thank you!

[rrbutani]

Looking into this again today; it seems like #151879 was actually a symptom of this issue (libstdc++ paths were being added when trying to use libc++, breaking include_next).

--gcc-toolchain is what causes clang to include these directories; --stdlib=libc++ inhibits this.

Both removing --gcc-toolchain or adding in --stdlib=libc++ are sufficient to fix this specific issue (i.e. compiling C++ code with clang++ with libc++).

As noted in cc-wrapper, dropping --gcc-toolchain is a little fraught and probably is not something we want to do when using libc++ because it also affects compiler-rt vs. libgcc_s:

nixpkgs/pkgs/build-support/cc-wrapper/default.nix

Lines 312 to 325 in d23c0da

	# TODO We would like to connect this to `useGccForLibs`, but we cannot yet
	# because `libcxxStdenv` on linux still needs this. Maybe someday we'll
	# always set `useLLVM` on Darwin, and maybe also break down `useLLVM` into
	# fine-grained use flags (libgcc vs compiler-rt, ld.lld vs legacy, libc++
	# vs libstdc++, etc.) since Darwin isn't `useLLVM` on all counts. (See
	# https://clang.llvm.org/docs/Toolchain.html for all the axes one might
	# break `useLLVM` into.)
	+ optionalString (isClang
	&& targetPlatform.isLinux
	&& !(stdenv.targetPlatform.useAndroidPrebuilt or false)
	&& !(stdenv.targetPlatform.useLLVM or false)
	&& gccForLibs != null) ''
	echo "--gcc-toolchain=${gccForLibs}" >> $out/nix-support/cc-cflags
	''

#151879 was fixed by #153963 which went the route of patching clang to add -nostdlibinc unconditionally to the command line arguments, effectively making it so that --gcc-toolchain does not influence the include directories at all (this doesn't break the clang++ + libstdc++ use case because cc-wrapper handles adding in the appropriate flags for that: 1, 2). This also has the additional upside that clang will be inhibited from searching FHS directories like /usr/lib for libstdc++ headers.

Unfortunately #153963 did have the side-effect of inadvertently breaking the clang + libstdc++ workflow (note: clang, not clang++, for compiling C++ code; this is, unfortunately, something that some build tools like bazel do; we get around this in nixpkgs but it seems better to just have the nixpkgs cc/c++ behave like they do elsewhere) because cc-wrapper only appends the libstdc++ flags to cxxflags. Previously, when using clang and libstdc++, clang would fill in those libstdc++ flags in for us (when building C++ code) via the --gcc-toolchain flag; passing in -nostdlibinc "broke" this.

(Note that the clang + libc++ flow was actually, amusingly, fixed by #153963; prior to the change, clang would fill in the libstdc++ dirs on top of the libc++ dirs because there was no -nostdlibinc to inhibit it from doing so. Unlike clang + libstdc++ the cc-wrapper machinery would effectively always add in the libc++ include and link dirs (via NIX_LDFLAGS) because of how they're added to cc.)
EDIT: this doesn't appear to be true; we need to have the include/c++/v1 dir included which only seems to be added here (i.e. for clang++: 1, 2, 3)

---

I think the bottom line is that we've got a matrix of toolchains and use cases we want to support (gcc or g++ or clang or clang++, libstdc++ or libc++) and a bunch of knobs we can turn (--stdlib, -nostdlibinc, --gcc-toolchain).

Also worth noting is that as of this writing LLVM 15 does not include the fix in #153963 though it probably should; I was mistaken and #151879 is observable with llvmPackages_15.libcxxStdenv though not when using clang instead of clang++ (because -nostdlibinc isn't present and because the include dir for libc++ with include/c++/v1 isn't present, clang ends up using libstdc++ in this case, incorrectly).

I think that:

• we do want the patch that implicitly applies -nostdlibinc (llvmPackages_13.clang: add nostdlibinc flag #153963) because it also prevents clang from searching /usr/lib (though there may be a way to achieve this with --sysroot instead)
• we do want to specify --gcc-toolchain even when using libc++ instead of libstdc++
• we should try to supply the C++ stdlib link and include flags (as well as -stdlib=libc++ as part of cflags) to both clang and clang++
  • I'm hoping this doesn't break any C code; hopefully there isn't any C code out there that's somehow banking on the compiler not having the C++ stdlib include directories
  • Another option is to mimic the language detection (based on file extension) that clang does and to set isCxx in cc-wrapper based on the files we see (but this seems fraught)

Ultimately cc-wrapper's desire to be in charge of managing libc and the C++ stdlib put it at odds with it's goal of mimicking the behavior of clang/clang++; clang will, upon detecting C++ source files, add in the C++ stdlib directories which cc-wrapper does not. I'm proposing we have cc-wrapper unconditionally include the C++ stdlib dirs, which, while still not an exact replication of clang's behavior, seems less likely to be observable in practice.

---

Apologies, this turned into a bit of a rant; I'll open a separate issue/PR for this later and try to figure out whether ^ is practical.