Bazel CI: remote execution is broken

[meteorcloudy]

https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/501#da3c2ee9-5c1d-4863-85bc-10a63dc700d0

ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/4e6a7dba926d8a5f2273fc776c7a457f/external/bazel_tools/tools/jdk/BUILD:197:1: Couldn't build file external/bazel_tools/tools/jdk/platformclasspath_classes: SkylarkAction external/bazel_tools/tools/jdk/platformclasspath_classes failed (Exit 2)
--
  | javac: directory not found: bazel-out/k8-fastbuild/bin/external/bazel_tools/tools/jdk/platformclasspath_classes
  | Usage: javac <options> <source files>
  | use -help for a list of possible options

Possible cause: 72ef398

@cushon Since the error message is related to java, so I guess it might be 72ef398 that broke it.
Can you help confirm this?

The culprit should be between dce9c45 and 72ef398

[nlopezgi]

This breakage seems to only affect RBE builds that use jdk8 as host_javabase, builds that use jdk10 are not affected. To get the build passing, please use:
--host_javabase=@bazel_toolchains_test_suite_repo//configs/ubuntu16_04_clang/1.1:jdk10
Also, make sure your build does not have the following (RBE recommended) flags:
--java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8 / --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8

Though this fixes the breakage, could @cushon please comment on whether we should now not expect to be able to compile Bazel with host_javabase pointing to a jdk8 (if so, it would be ideal to try to get this behind a --incompatible_xxx flag so we can follow breaking change policy?).

[meteorcloudy]

FYI @philwo

[cushon]

The culprit is a392d0e, which I'm rolling back.

This breakage seems to only affect RBE builds that use jdk8 as host_javabase

That's consistent with the original problem. There's an action that runs the host_javabase's javac with a TreeArtifact as the output directory. The JDK 8 javac requires the output directory to exist, while JDK 9 will create the output directory.

Can someone confirm that the bug still occurs with --strategy=remote using the minimal repro? (or point me at documentation to do that?)
#6262 (comment)

[cushon]

I reproduced the remote execution bug and filed #6393.