rules_python build_file_generation integration tests on Ubuntu using minimum supported Bazel version is failing with "requirements_test on Bazel@HEAD #1520

sgowroji · 2023-10-30T07:04:19Z

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3423#018b7e52-e06d-4e38-8577-97686d10d343

Platform : rules_python (build_file_generation integration tests on Ubuntu)

Logs:


(02:05:20) FAIL: //:requirements_test (see /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6a1011ee018bdbf38289288c626fc42e/execroot/_main/bazel-out/k8-fastbuild/testlogs/requirements_test/test.log)
 
FAILED: //:requirements_test (Summary)

Steps:

git clone -v https://github.com/bazelbuild/rules_python.git
git reset 5b082bb02b048a4b630843b6b582bd6f40d38b9e --hard
export USE_BAZEL_VERSION=8cab6aa21fa765985a962e59264e12251e4d479f
bazel test -- ...

Autosheriff culprit : bazelbuild/bazel@32cc97e

CC Greenteam @comius

The text was updated successfully, but these errors were encountered:

… tests The Bazel downstream tests will use Bazel built at head, but the tests checking for support with the minimum Bazel version are specifically intended to only run with an older Bazel version. Work towards bazelbuild#1520

rickeylev · 2023-10-30T21:49:44Z

qq: did the system python version change on the build hosts? i.e. from python 3.8 to python 3.9? That looks to be a common diff showing up. (I'm not asking to undo that, it would just explain why this failure shows up).

PR 1522 it out to partially address this -- many of those tests are trying to run bazel at head for the configs that should only be running Bazel 5.4. That will, at the least, skip those configs where it doesn't need to run.

I'm still looking into the root cause.

rickeylev · 2023-10-30T21:59:57Z

Ah, I think this is because of switching bzlmod on by default. Setting --noenable_bzlmod when running some of the failing tests fixes it.

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Fixes bazelbuild#1520

#1522) The Bazel downstream tests will use Bazel built at head, but the tests checking for support with the minimum Bazel version are specifically intended to only run with an older Bazel version. Work towards #1520

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Work towards #1520

Bazel at head has bzlmod enabled by default, so the example needs to be updated to work with bzlmod enabled. Work towards bazelbuild#1520

… with bzlmod Bazel at head enables bzlmod by default, so the tests must also be updated to be bzlmod compatible. Also makes compile_pip_requirements ignore its convenience symlinks. This allows locally running the "from external" workspace test even if the compile_pip_requirements workspace previously had convenience symlinks in it from other builds. Work towards bazelbuild#1520

rickeylev · 2023-10-31T22:09:37Z

Ok, I think I have PRs out for all the failures shown in that link. They're all fairly simple fixes. Most just need a MODULE.bazel file, a couple needed to change how they were finding files in runfiles.

Bazel at head has bzlmod enabled by default, so the example needs to be updated to work with bzlmod enabled. Work towards #1520

… with bzlmod Bazel at head enables bzlmod by default, so the tests must also be updated to be bzlmod compatible. Also makes compile_pip_requirements ignore its convenience symlinks. This allows locally running the "from external" workspace test even if the compile_pip_requirements workspace previously had convenience symlinks in it from other builds. Work towards bazelbuild#1520

… with bzlmod (#1528) Bazel at head enables bzlmod by default, so the tests must also be updated to be bzlmod compatible. Also makes compile_pip_requirements ignore its convenience symlinks. This allows locally running the "from external" workspace test even if the compile_pip_requirements workspace previously had convenience symlinks in it from other builds. Work towards #1520

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Work towards #1520

rickeylev · 2023-11-01T16:00:36Z

All the PRs are merged. I think that fixes all the failures. @sgowroji let us know if more failures show up in subsequent downstream bazel runs

sgowroji · 2023-11-06T05:04:16Z

Hi @rickeylev, Rules_Python is still failing with below errors with Bazel@HEAD.
build_file_generation integration tests on Ubuntu
example bzlmod build file integration test
pip_repository_annotations integration tests on Ubuntu

The build_file_generation example only supports workspace style builds. The bzlmod equivalent is in `examples/bzlmod_build_file_generation` Work towards bazelbuild#1520

rickeylev · 2023-11-06T22:10:13Z

Thanks. Looks like the same cause: bzlmod enabled by default.

…xample The pip_repository_annoations example is workspace-only. Its equivalent functionality is demonstrated in examples/bzlmod with the whl_mods feature. Work towards bazelbuild#1520

Part of the pyc compilation process is to create a temporary file named `<name>.pyc.NNNN`, where `NNNN` is a timestamp. Once the pyc is entirely written, this file is renamed to the regular pyc file name. These files only exist for brief periods of time, but its possible for different threads/processes to see the temporary files when computing the glob() values. Later, since the file is gone, an error is raised about the file missing. PR bazelbuild#1266 mostly fixed this issue, except that the glob exclude for an interpreter runtime's files was behind the `ignore_root_user_error` flag, which meant it wasn't always applied. This changes it to always be applied, which should eliminate the failures. Fixes bazelbuild#1261 Work towards bazelbuild#1520

…xample (#1540) The pip_repository_annoations example is workspace-only. Its equivalent functionality is demonstrated in examples/bzlmod with the whl_mods feature. Work towards #1520

#1539) The build_file_generation example only supports workspace style builds. The bzlmod equivalent is in `examples/bzlmod_build_file_generation` Work towards #1520

…1541) Part of the pyc compilation process is to create a temporary file named `<name>.pyc.NNNN`, where `NNNN` is a timestamp. Once the pyc is entirely written, this file is renamed to the regular pyc file name. These files only exist for brief periods of time, but its possible for different threads/processes to see the temporary files when computing the glob() values. Later, since the file is gone, an error is raised about the file missing. PR #1266 mostly fixed this issue, except that the exclude for the `.pyc.NNNN` files for an interpreter runtime's files was behind the `ignore_root_user_error` flag, which meant it wasn't always applied. This changes it to always be applied, which should eliminate the failures due to the missing NNNN files. Fixes #1261 Work towards #1520

rickeylev · 2023-11-07T22:42:53Z

Ok, all PRs merged. Fixed 🤞

rickeylev mentioned this issue Oct 30, 2023

ci: don't run minimum bazel version tests as part of bazel downstream… #1522

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Oct 30, 2023

tests: make multi_python_verions example bzlmod compatible

bef9d39

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Fixes bazelbuild#1520

rickeylev mentioned this issue Oct 30, 2023

tests: make multi_python_verions example bzlmod compatible #1523

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Oct 30, 2023

tests: make multi_python_verions example bzlmod compatible

103cbee

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Fixes bazelbuild#1520

rickeylev mentioned this issue Oct 30, 2023

tests: make pip_parse example work with bzlmod enabled #1524

Merged

github-merge-queue bot pushed a commit that referenced this issue Oct 31, 2023

tests: make multi_python_verions example bzlmod compatible (#1523)

843084d

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Work towards #1520

rickeylev mentioned this issue Oct 31, 2023

test: make compile_pip_requirements work with bzlmod enabled #1526

Merged

rickeylev mentioned this issue Oct 31, 2023

test: make compile_pip_requirements_test_from_external_workspace work with bzlmod #1528

Merged

github-merge-queue bot pushed a commit that referenced this issue Oct 31, 2023

test: make compile_pip_requirements work with bzlmod enabled (#1526)

a2789cc

Bazel at head has bzlmod enabled by default, so the example needs to be updated to work with bzlmod enabled. Work towards #1520

github-merge-queue bot pushed a commit that referenced this issue Nov 1, 2023

tests: make pip_parse example work with bzlmod enabled (#1524)

821a323

Bazel is enabling bzlmod by default, which means the examples need to be updated to be bzlmod compatible. Work towards #1520

rickeylev closed this as completed Nov 1, 2023

rickeylev mentioned this issue Nov 6, 2023

tests: disable bzlmod for workspace-only build_file_generation example #1539

Merged

rickeylev reopened this Nov 6, 2023

rickeylev mentioned this issue Nov 6, 2023

tests: disable bzlmod for workspace-only pip_repository_annotations example #1540

Merged

rickeylev mentioned this issue Nov 6, 2023

fix: always ignore .pyc.NNNN files from the hermetic runtime tree #1541

Merged

rickeylev closed this as completed Nov 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rules_python build_file_generation integration tests on Ubuntu using minimum supported Bazel version is failing with "requirements_test on Bazel@HEAD #1520

rules_python build_file_generation integration tests on Ubuntu using minimum supported Bazel version is failing with "requirements_test on Bazel@HEAD #1520

sgowroji commented Oct 30, 2023 •

edited

Loading

rickeylev commented Oct 30, 2023

rickeylev commented Oct 30, 2023

rickeylev commented Oct 31, 2023

rickeylev commented Nov 1, 2023

sgowroji commented Nov 6, 2023

rickeylev commented Nov 6, 2023

rickeylev commented Nov 7, 2023

rules_python build_file_generation integration tests on Ubuntu using minimum supported Bazel version is failing with "requirements_test on Bazel@HEAD #1520

rules_python build_file_generation integration tests on Ubuntu using minimum supported Bazel version is failing with "requirements_test on Bazel@HEAD #1520

Comments

sgowroji commented Oct 30, 2023 • edited Loading

rickeylev commented Oct 30, 2023

rickeylev commented Oct 30, 2023

rickeylev commented Oct 31, 2023

rickeylev commented Nov 1, 2023

sgowroji commented Nov 6, 2023

rickeylev commented Nov 6, 2023

rickeylev commented Nov 7, 2023

sgowroji commented Oct 30, 2023 •

edited

Loading