Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: use free runners for x86_64-gnu-llvm jobs #134124

Merged
merged 21 commits into from
Dec 16, 2024
8 changes: 6 additions & 2 deletions src/ci/docker/host-x86_64/x86_64-gnu-llvm-18/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -59,5 +59,9 @@ COPY host-x86_64/dist-x86_64-linux/build-gccjit.sh /scripts/

RUN /scripts/build-gccjit.sh /scripts

COPY scripts/x86_64-gnu-llvm.sh /tmp/script.sh
ENV SCRIPT /tmp/script.sh
ARG SCRIPT_ARG
COPY scripts/x86_64-gnu-llvm.sh /tmp/x86_64-gnu-llvm.sh
COPY scripts/x86_64-gnu-llvm1.sh /tmp/x86_64-gnu-llvm1.sh
COPY scripts/x86_64-gnu-llvm2.sh /tmp/x86_64-gnu-llvm2.sh
COPY scripts/x86_64-gnu-llvm3.sh /tmp/x86_64-gnu-llvm3.sh
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't only copy the right file because this line wants the exact path (I tried to use ${SCRIPT_ARG} and it failed.

ENV SCRIPT /tmp/${SCRIPT_ARG}
8 changes: 6 additions & 2 deletions src/ci/docker/host-x86_64/x86_64-gnu-llvm-19/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -59,5 +59,9 @@ COPY host-x86_64/dist-x86_64-linux/build-gccjit.sh /scripts/

RUN /scripts/build-gccjit.sh /scripts

COPY scripts/x86_64-gnu-llvm.sh /tmp/script.sh
ENV SCRIPT /tmp/script.sh
ARG SCRIPT_ARG
COPY scripts/x86_64-gnu-llvm.sh /tmp/x86_64-gnu-llvm.sh
COPY scripts/x86_64-gnu-llvm1.sh /tmp/x86_64-gnu-llvm1.sh
COPY scripts/x86_64-gnu-llvm2.sh /tmp/x86_64-gnu-llvm2.sh
COPY scripts/x86_64-gnu-llvm3.sh /tmp/x86_64-gnu-llvm3.sh
ENV SCRIPT /tmp/${SCRIPT_ARG}
19 changes: 19 additions & 0 deletions src/ci/docker/scripts/add_dummy_commit.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/bash

set -ex

if [ "$READ_ONLY_SRC" = "0" ]; then
# `core::builder::tests::ci_rustc_if_unchanged_logic` bootstrap test ensures that
# "download-rustc=if-unchanged" logic don't use CI rustc while there are changes on
# compiler and/or library. Here we are adding a dummy commit on compiler and running
# that test to make sure we never download CI rustc with a change on the compiler tree.
echo "" >> ../compiler/rustc/src/main.rs
git config --global user.email "dummy@dummy.com"
git config --global user.name "dummy"
git add ../compiler/rustc/src/main.rs
git commit -m "test commit for rust.download-rustc=if-unchanged logic"
DISABLE_CI_RUSTC_IF_INCOMPATIBLE=0 ../x.py test bootstrap \
-- core::builder::tests::ci_rustc_if_unchanged_logic
# Revert the dummy commit
git reset --hard HEAD~1
fi
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extracted from x86_64-gnu-llvm.sh since it's common to all scripts

37 changes: 1 addition & 36 deletions src/ci/docker/scripts/x86_64-gnu-llvm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,7 @@

set -ex

if [ "$READ_ONLY_SRC" = "0" ]; then
# `core::builder::tests::ci_rustc_if_unchanged_logic` bootstrap test ensures that
# "download-rustc=if-unchanged" logic don't use CI rustc while there are changes on
# compiler and/or library. Here we are adding a dummy commit on compiler and running
# that test to make sure we never download CI rustc with a change on the compiler tree.
echo "" >> ../compiler/rustc/src/main.rs
git config --global user.email "dummy@dummy.com"
git config --global user.name "dummy"
git add ../compiler/rustc/src/main.rs
git commit -m "test commit for rust.download-rustc=if-unchanged logic"
DISABLE_CI_RUSTC_IF_INCOMPATIBLE=0 ../x.py test bootstrap \
-- core::builder::tests::ci_rustc_if_unchanged_logic
# Revert the dummy commit
git reset --hard HEAD~1
fi

# Only run the stage 1 tests on merges, not on PR CI jobs.
if [[ -z "${PR_CI_JOB}" ]]; then
../x.py --stage 1 test --skip src/tools/tidy

# Run the `mir-opt` tests again but this time for a 32-bit target.
# This enforces that tests using `// EMIT_MIR_FOR_EACH_BIT_WIDTH` have
# both 32-bit and 64-bit outputs updated by the PR author, before
# the PR is approved and tested for merging.
# It will also detect tests lacking `// EMIT_MIR_FOR_EACH_BIT_WIDTH`,
# despite having different output on 32-bit vs 64-bit targets.
../x.py --stage 1 test tests/mir-opt --host='' --target=i686-unknown-linux-gnu

# Run `ui-fulldeps` in `--stage=1`, which actually uses the stage0
# compiler, and is sensitive to the addition of new flags.
../x.py --stage 1 test tests/ui-fulldeps

# Rebuild the stdlib with the size optimizations enabled and run tests again.
RUSTFLAGS_NOT_BOOTSTRAP="--cfg feature=\"optimize_for_size\"" ../x.py --stage 1 test \
library/std library/alloc library/core
fi
./add_dummy_commit.sh

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this because this script only runs in PR

# NOTE: intentionally uses all of `x.py`, `x`, and `x.ps1` to make sure they all work on Linux.
../x.py --stage 2 test --skip src/tools/tidy
Expand Down
12 changes: 12 additions & 0 deletions src/ci/docker/scripts/x86_64-gnu-llvm1.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash

set -ex

./add_dummy_commit.sh

../x.py --stage 2 test \
--skip tests \
--skip coverage-map \
--skip coverage-run \
--skip library \
--skip tidyselftest
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are duplicated from the CI sets.

One way to de-duplicate them might be to move the CI anchors to scripts.
We could do this in another PR. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if the scripts could encompass everything, e.g. the Windows runners don't use bash scripts within Docker, some runners still use a Makefile.. Let's leave it like this for now.

25 changes: 25 additions & 0 deletions src/ci/docker/scripts/x86_64-gnu-llvm2.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash

set -ex

./add_dummy_commit.sh

##### Test stage 2 #####

../x.py --stage 2 test \
--skip compiler \
--skip src

# Run the `mir-opt` tests again but this time for a 32-bit target.
# This enforces that tests using `// EMIT_MIR_FOR_EACH_BIT_WIDTH` have
# both 32-bit and 64-bit outputs updated by the PR author, before
# the PR is approved and tested for merging.
# It will also detect tests lacking `// EMIT_MIR_FOR_EACH_BIT_WIDTH`,
# despite having different output on 32-bit vs 64-bit targets.
../x --stage 2 test tests/mir-opt --host='' --target=i686-unknown-linux-gnu

# Run the UI test suite again, but in `--pass=check` mode
#
# This is intended to make sure that both `--pass=check` continues to
# work.
../x.ps1 --stage 2 test tests/ui --pass=check --host='' --target=i686-unknown-linux-gnu
25 changes: 25 additions & 0 deletions src/ci/docker/scripts/x86_64-gnu-llvm3.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash

set -ex

./add_dummy_commit.sh

##### Test stage 1 #####

../x.py --stage 1 test --skip src/tools/tidy

# Run the `mir-opt` tests again but this time for a 32-bit target.
# This enforces that tests using `// EMIT_MIR_FOR_EACH_BIT_WIDTH` have
# both 32-bit and 64-bit outputs updated by the PR author, before
# the PR is approved and tested for merging.
# It will also detect tests lacking `// EMIT_MIR_FOR_EACH_BIT_WIDTH`,
# despite having different output on 32-bit vs 64-bit targets.
../x.py --stage 1 test tests/mir-opt --host='' --target=i686-unknown-linux-gnu

# Run `ui-fulldeps` in `--stage=1`, which actually uses the stage0
# compiler, and is sensitive to the addition of new flags.
../x.py --stage 1 test tests/ui-fulldeps

# Rebuild the stdlib with the size optimizations enabled and run tests again.
RUSTFLAGS_NOT_BOOTSTRAP="--cfg feature=\"optimize_for_size\"" ../x.py --stage 1 test \
library/std library/alloc library/core
51 changes: 47 additions & 4 deletions src/ci/github-actions/jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ pr:
ENABLE_GCC_CODEGEN: "1"
# We are adding (temporarily) a dummy commit on the compiler
READ_ONLY_SRC: "0"
DOCKER_SCRIPT: x86_64-gnu-llvm.sh
<<: *job-linux-16c
- image: x86_64-gnu-tools
<<: *job-linux-16c
Expand Down Expand Up @@ -312,16 +313,58 @@ auto:
- image: x86_64-gnu-distcheck
<<: *job-linux-8c

- image: x86_64-gnu-llvm-19
# The x86_64-gnu-llvm-19 job is split into multiple jobs to run tests in parallel.
# x86_64-gnu-llvm-19-1 skips tests that run in x86_64-gnu-llvm-19-{2,3}.
- image: x86_64-gnu-llvm-19-1
env:
RUST_BACKTRACE: 1
<<: *job-linux-8c
IMAGE: x86_64-gnu-llvm-19
DOCKER_SCRIPT: x86_64-gnu-llvm1.sh
<<: *job-linux-4c

- image: x86_64-gnu-llvm-18
# Skip tests that run in x86_64-gnu-llvm-19-{1,3}
- image: x86_64-gnu-llvm-19-2
env:
RUST_BACKTRACE: 1
IMAGE: x86_64-gnu-llvm-19
DOCKER_SCRIPT: x86_64-gnu-llvm2.sh
<<: *job-linux-4c

# Skip tests that run in x86_64-gnu-llvm-19-{1,2}
- image: x86_64-gnu-llvm-19-3
env:
RUST_BACKTRACE: 1
IMAGE: x86_64-gnu-llvm-19
DOCKER_SCRIPT: x86_64-gnu-llvm3.sh
<<: *job-linux-4c

# The x86_64-gnu-llvm-18 job is split into multiple jobs to run tests in parallel.
# x86_64-gnu-llvm-18-1 skips tests that run in x86_64-gnu-llvm-18-{2,3}.
- image: x86_64-gnu-llvm-18-1
env:
RUST_BACKTRACE: 1
READ_ONLY_SRC: "0"
<<: *job-linux-8c
IMAGE: x86_64-gnu-llvm-18
DOCKER_SCRIPT: x86_64-gnu-llvm1.sh
<<: *job-linux-4c

# Skip tests that run in x86_64-gnu-llvm-18-{1,3}
- image: x86_64-gnu-llvm-18-2
env:
RUST_BACKTRACE: 1
READ_ONLY_SRC: "0"
IMAGE: x86_64-gnu-llvm-18
DOCKER_SCRIPT: x86_64-gnu-llvm2.sh
<<: *job-linux-4c

# Skip tests that run in x86_64-gnu-llvm-18-{1,2}
- image: x86_64-gnu-llvm-18-3
env:
RUST_BACKTRACE: 1
READ_ONLY_SRC: "0"
IMAGE: x86_64-gnu-llvm-18
DOCKER_SCRIPT: x86_64-gnu-llvm3.sh
<<: *job-linux-4c

- image: x86_64-gnu-nopt
<<: *job-linux-4c
Expand Down
Loading