Cannot build candidate for 0.24.0: undeclared inclusions in protobuf

[katre]

Logs at https://buildkite.com/bazel-trusted/bazel-release/builds/13#d4abc10f-31bb-4b37-adab-4e921d302477

Error message:

ERROR: /home/bazel/.cache/bazel/_bazel_bazel/ec321eb2cc2d0f8f91b676b6d4c66c29/external/io_bazel/third_party/protobuf/3.6.1/BUILD:123:1: undeclared inclusion(s) in rule '@io_bazel//third_party/protobuf/3.6.1:protobuf_lite':
  | this rule is missing dependency declarations for the following files included by 'external/io_bazel/third_party/protobuf/3.6.1/src/google/protobuf/stubs/statusor.cc':
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/statusor.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/status.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/common.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/port.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/platform_macros.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/macros.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/logging.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/mutex.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/callback.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/stringpiece.h'
  | '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/hash.h'
 ```

[katre]

I can't reproduce this locally, but it's consistent on BuildKite.

@tetromino, do you know anything from the protobuf side?
@lberki or @hlopko do you know anything from the cc library side?

[katre]

Oh, and I don't seem to have the required access to download the docker image and reproduce this directly, @fweikert do you know how I can get that?

[tetromino]

//third_party/protobuf/3.6.1 uses angle-bracket includes everywhere. For example, in statusor.cc: #include <google/protobuf/stubs/statusor.h>

Is it possible that the include path is such that something (either the compiler or the include scanner) is pulling in includes from the wrong place - from /workdir/third_party/protobuf/3.6.1 insted of the expected place (I'm guessing something like /workdir/external/io_bazel/third_party/protobuf/3.6.1)?

Maybe replacing angle-brackets with "" would help...

[katre]

Asking CI team to take a look: @meteorcloudy @fweikert @philwo

[katre]

Adding the full log of the failing build:
bazel-release_build_13_ubuntu-14-dot-04.log

[philwo]

@katre I can reproduce this locally in Docker:

# On my workstation:
cd $HOME/src
git clone https://github.com/bazelbuild/bazel.git bazel-for-docker
sudo chown -R 999:999 bazel-for-docker
docker run -it --rm --init --volume $HOME/src/bazel-for-docker:/workdir --workdir /workdir --network host gcr.io/bazel-public/ubuntu1404:java8 /bin/bash

# Now inside the Docker container:
git fetch --force origin master
git fetch --force origin refs/notes/*:refs/notes/*
bazel build //src:bazel
mkdir output
cp bazel-bin/src/bazel output/bazel
output/bazel build -c opt --stamp --embed_label "0.24.0rc1" --workspace_status_command=scripts/ci/build_status_command.sh src/bazel scripts/packages/with-jdk/install.sh scripts/packages/debian/bazel-debian.deb scripts/packages/debian/bazel.dsc scripts/packages/debian/bazel.tar.gz bazel-distfile.zip

# Results in:
[...]
ERROR: /home/bazel/.cache/bazel/_bazel_bazel/ec321eb2cc2d0f8f91b676b6d4c66c29/external/io_bazel/third_party/protobuf/3.6.1/BUILD:123:1: undeclared inclusion(s) in rule '@io_bazel//third_party/protobuf/3.6.1:protobuf_lite':
this rule is missing dependency declarations for the following files included by 'external/io_bazel/third_party/protobuf/3.6.1/src/google/protobuf/stubs/bytestream.cc':
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/bytestream.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/common.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/port.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/platform_macros.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/macros.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/logging.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/mutex.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/callback.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/stringpiece.h'
  '/workdir/third_party/protobuf/3.6.1/src/google/protobuf/stubs/hash.h'
INFO: Elapsed time: 44.423s, Critical Path: 22.00s
INFO: 1110 processes: 1 local, 1109 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

I think this is a bug in Bazel.

[philwo]

Automatic bisecting:

git bisect start
git checkout master
bazel build //src:bazel && cp -f bazel-bin/src/bazel output/bazel && output/bazel build -c opt --stamp --embed_label 0.24.0rc1 --workspace_status_command=scripts/ci/build_status_command.sh src/bazel scripts/packages/with-jdk/install.sh scripts/packages/debian/bazel-debian.deb scripts/packages/debian/bazel.dsc scripts/packages/debian/bazel.tar.gz bazel-distfile.zip
# This failed.
git bisect bad

# Checkout baseline of 0.23.0, which should still be fine.
git checkout 441fd75d0047f8a998d784c557736ab9075db893
bazel build //src:bazel && cp -f bazel-bin/src/bazel output/bazel && output/bazel build -c opt --stamp --embed_label 0.24.0rc1 --workspace_status_command=scripts/ci/build_status_command.sh src/bazel scripts/packages/with-jdk/install.sh scripts/packages/debian/bazel-debian.deb scripts/packages/debian/bazel.dsc scripts/packages/debian/bazel.tar.gz bazel-distfile.zip
# That worked.
git bisect good

# Now run the automatic bisect:
git bisect run sh -c "bazel build //src:bazel && cp -f bazel-bin/src/bazel output/bazel && output/bazel build -c opt --stamp --embed_label 0.24.0rc1 --workspace_status_command=scripts/ci/build_status_command.sh src/bazel scripts/packages/with-jdk/install.sh scripts/packages/debian/bazel-debian.deb scripts/packages/debian/bazel.dsc scripts/packages/debian/bazel.tar.gz bazel-distfile.zip"

The culprit is this one: 1b4c37c

Verification:

git checkout master
git config --global user.email philwo@google.com
git config --global user.name 'Philipp Wollermann'
git revert 1b4c37c38804559b5c1ade6f9c93501875e231b0
bazel build //src:bazel && cp -f bazel-bin/src/bazel output/bazel && output/bazel build -c opt --stamp --embed_label 0.24.0rc1 --workspace_status_command=scripts/ci/build_status_command.sh src/bazel scripts/packages/with-jdk/install.sh scripts/packages/debian/bazel-debian.deb scripts/packages/debian/bazel.dsc scripts/packages/debian/bazel.tar.gz bazel-distfile.zip

# Success:
[...]
INFO: Elapsed time: 154.705s, Critical Path: 79.11s
INFO: 2871 processes: 2 local, 2394 processwrapper-sandbox, 475 worker.
INFO: Build completed successfully, 2949 total actions

@katre I recommend to rollback the culprit and cherry-pick that into 0.24.0.

[asuffield]

That's annoying. Does anybody know exactly what the bad behaviour is from --no-canonical-prefixes here, or do we still need to debug it?

It is frustratingly difficult to precisely identify the situations in which this flag must be set and must not be set.

[hlopko]

Hi all, I'm creating a rollback so we can unblock 0.24. What the actual problem is still needs investigation. @asuffield, will you have capacity to pursue this?

[katre]

Thanks for the rollback @hlopko. I will cherry-pick that into the release and if it works I will close this issue.

[katre]

This is now resolved. Thanks for the help everyone.