Sandboxed execution breaks Kythe build

[schroederc]

As per e0ac088, here's a bug filed about Kythe's build breaking due to the --spawn_strategy and --genrule_strategy default value changes.

[damienmg]

Can you add more details?

[philwo]

Can you give me instructions how to reproduce the build failure and the error message?

[schroederc]

Building Kythe instructions: http://www.kythe.io/contributing#building-source-code

Remove the --spawn_strategy=standalone --genrule_strategy=standalone options:

rm tools/bazel.rc

Example error messages:

bazel test //kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg:names_tests
INFO: Writing profile data to '/tmp/bazel.prof'
INFO: Found 1 test target...
INFO: Writing explanation of rebuilds to '/tmp/bazel.explain'
ERROR: Sandboxed execution failed: //kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg:names_tests..
INFO: From MockJavac kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg/names_tests.jar:
/bin/bash: line 3: external/local-jdk/bin/javac: No such file or directory

bazel build //third_party/go:protobuf
INFO: Writing profile data to '/tmp/bazel.prof'
INFO: Found 1 target...
INFO: Writing explanation of rebuilds to '/tmp/bazel.explain'
ERROR: Sandboxed execution failed: //third_party/go:protobuf..
INFO: From GoCompile third_party/go/protobuf.a:
go tool: no tool directory: open /usr/local/go/pkg/tool/linux_amd64: no such file or directory

bazel build //tools/modules:check_modules
INFO: Writing profile data to '/tmp/bazel.prof'
INFO: Found 1 target...
INFO: Writing explanation of rebuilds to '/tmp/bazel.explain'
ERROR: Sandboxed execution failed: //tools/modules:check_modules..
INFO: From Executing genrule //tools/modules:check_modules:
Missing LLVM with ancestor 8d1daf644b73430ef077b13e94ac4ef78abbe238 (rev 242330) in third_party/llvm/llvm
Please run //tools/modules/update.sh.
tools/modules/check.sh: line 27: cd: third_party/llvm/llvm: No such file or directory
tools/modules/check.sh: line 30: cd: third_party/llvm/llvm: No such file or director

[philwo]

I'm trying to reproduce this, but that tools/modules/update.sh script that I'm supposed to run before "bazel build //..." is building a lot of .cpp files with llvm since over an hour already :| On a 4-CPU 16GB SSD GCE instance...

[philwo]

That script never finished, but it turned out that the bazel build parts work fine without it. :)

These are all errors caused by different reasons:

1. bazel test //kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg:names_tests

In your tools/build_rules/kythe.bzl, you shouldn't hardcode the path to external/local-jdk/bin/javac, but instead resolve that via javac's label. Also javac + the JDK is missing from the action inputs, which is why you get the "file not found" error for javac when running in the sandbox. Same for "jar", which would fail right after javac, as it's also not included in the inputs.

Check our experimental Java Skylark rules for ideas how to do this:
https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/java_rules_skylark.bzl

My commit here basically fixed the same issues in our own rules, I think you can just adapt it (the parts that affect the .bzl file):
c8db70c

2. bazel build //third_party/go:protobuf
   The error message that you posted looks like the same reason - compiler missing in the inputs of the action. However, I got a different one, that was actually caused by a bug in the sandbox code: We didn't copy the symbolic link that you generate as an output, because the code only worked on regular files. I'm sending a fix for that ASAP.

3. bazel build //tools/modules:check_modules
   Same here - if you look at the genrule, you can see that the only inputs that the sandbox will mount for it are "check.sh" and "versions.sh", because no others are specified in the rule. But your check.sh script is accessing llvm - so that has to be in the inputs, too! I think you could create a filegroup in the llvm/BUILD file that includes the files you need for the check.sh script and put that into the "tools" (or srcs, not sure) attribute of the genrule.

[ulfjack]

In the tools attribute.

[schroederc]

I've been fixing some of the build issues. Along the way, I ran into the symlink issue, and I also ran into the following error:

ERROR: Sandboxed execution failed: //tools/modules:check_modules..
ERROR: /usr/local/google/home/schroederc/kythe/tools/modules/BUILD:9:1: Couldn't build file tools/modules/modules_checked_empty.cc: Executing genrule //tools/modules:check_modules failed: Error during execution of spawn: java.io.IOException: Cannot run program "/usr/local/google/home/schroederc/.cache/bazel/_bazel_schroederc/ddace01a1ad38df359eb57115be59fad/kythe/_bin/namespace-sandbox" (in directory "/usr/local/google/home/schroederc/.cache/bazel/_bazel_schroederc/ddace01a1ad38df359eb57115be59fad/kythe"): error=7, Argument list too long.

Any hack around this kind of issue? Can this limit be increased?

[schroederc]

Also, --verbose_failures is nearly unusable when debugging these sandbox errors. There are TONS of mount lines.

[bsilver8192]

FYI, I filed #424 about the --verbose_failures issue.

[philwo]

Any hack around this kind of issue? Can this limit be increased?

Yes, I'll implement our standard Bazel hack for this, which is to write the arguments into a file and then pass the name of that file prefixed with "@" to the namespace-runner.

[philwo]

@schroederc Can you check whether this commit already helped a bit regarding the "Argument list too long" issue?

17ef915

It should cut the amount of command-line parameters passed to the sandbox runner in half on average (because the mount target is now inferred from the source path, unless a custom target path is necessary - this should only be the case for runfiles and never for inputs).

[philwo]

Just tried the three test cases from above again and //third_party/go:protobuf and //tools/modules:check_modules work fine for me now.

bazel test //kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg:names_tests

fails with:

philwo@philwo-bazel-debian81:~/kythe$ bazel test //kythe/javatests/com/google/devtools/kythe/analyzers/java/testdata/pkg:names_tests
INFO: Found 1 test target...
INFO: From GoCompile third_party/go/protoc-gen-go_descriptor.a:
third_party/go/src/github.com/golang/protobuf/protoc-gen-go/descriptor/descriptor.pb.go:33: can't find import: "github.com/golang/protobuf/proto"
ERROR: /home/philwo/kythe/third_party/go/BUILD:119:1: error executing shell command: 'set -e

The missing package that it can't find seems to be in:
bazel-out/local_linux-fastbuild/bin/third_party/go/protobuf.a_gopath

I think this is because in tools/build_rules/go.bzl you construct the path to the include:
line 46: include_paths += "-I "" + dep.go_archive.path + "_gopath" "

but that "dep.go_archive.path + '_gopath'" directory is not in the inputs of the action.

Indeed, you can see via --sandbox_debug that the archive itself is being mounted, but not the _gopath directory:

src/main/tools/namespace-sandbox.c:393: mount: /home/philwo/.cache/bazel/_bazel_philwo/0a87a7957485c05598ee2035800ac806/kythe/bazel-out/local_linux-fastbuild/bin/third_party/go/protobuf.a

The _gopath directory comes from the symlink created in the "if setupGOPATH" part in line 78 and 114. I think you can just add it to the outputs of the go_library definition and then pass it to the inputs of the action that depend on it in the go_compile method.

Maybe this helps in fixing the issue? Is there anything else remaining?

[hanwen]

seems that this is not a bazel issue, so I'll close.