Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[do not merge] Testing Windows CI #127373

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 82 additions & 67 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@ jobs:
with:
fetch-depth: 2

- run: Set-MpPreference -DisableRealtimeMonitoring $true
shell: powershell
- run: Get-MpComputerStatus
shell: powershell

# Rust Log Analyzer can't currently detect the PR number of a GitHub
# Actions build on its own, so a hint in the log message is needed to
# point it in the right direction.
Expand All @@ -125,44 +130,44 @@ jobs:
- name: ensure the channel matches the target branch
run: src/ci/scripts/verify-channel.sh

- name: collect CPU statistics
run: src/ci/scripts/collect-cpu-stats.sh
# - name: collect CPU statistics
# run: src/ci/scripts/collect-cpu-stats.sh

- name: show the current environment
run: src/ci/scripts/dump-environment.sh

- name: install awscli
run: src/ci/scripts/install-awscli.sh
# - name: install awscli
# run: src/ci/scripts/install-awscli.sh

- name: install sccache
run: src/ci/scripts/install-sccache.sh

- name: select Xcode
run: src/ci/scripts/select-xcode.sh
# - name: select Xcode
# run: src/ci/scripts/select-xcode.sh

- name: install clang
run: src/ci/scripts/install-clang.sh

- name: install tidy
run: src/ci/scripts/install-tidy.sh
# - name: install tidy
# run: src/ci/scripts/install-tidy.sh

- name: install WIX
run: src/ci/scripts/install-wix.sh
# - name: install WIX
# run: src/ci/scripts/install-wix.sh

- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh

- name: checkout submodules
run: src/ci/scripts/checkout-submodules.sh

- name: install MinGW
run: src/ci/scripts/install-mingw.sh
# - name: install MinGW
# run: src/ci/scripts/install-mingw.sh

- name: install ninja
run: src/ci/scripts/install-ninja.sh

- name: enable ipv6 on Docker
run: src/ci/scripts/enable-docker-ipv6.sh
# - name: enable ipv6 on Docker
# run: src/ci/scripts/enable-docker-ipv6.sh

# Disable automatic line ending conversion (again). On Windows, when we're
# installing dependencies, something switches the git configuration directory or
Expand All @@ -172,14 +177,20 @@ jobs:
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh

- name: ensure line endings are correct
run: src/ci/scripts/verify-line-endings.sh
# - name: ensure line endings are correct
# run: src/ci/scripts/verify-line-endings.sh

# - name: ensure backported commits are in upstream branches
# run: src/ci/scripts/verify-backported-commits.sh

- name: ensure backported commits are in upstream branches
run: src/ci/scripts/verify-backported-commits.sh
# - name: ensure the stable version number is correct
# run: src/ci/scripts/verify-stable-version-number.sh

- name: ensure the stable version number is correct
run: src/ci/scripts/verify-stable-version-number.sh
- name: Download Handle
run: |
curl -O https://download.sysinternals.com/files/Handle.zip
unzip -d handle Handle.zip
echo "$PWD/handle" >> $GITHUB_PATH

- name: run the build
# Redirect stderr to stdout to avoid reordering the two streams in the GHA logs.
Expand All @@ -188,53 +199,57 @@ jobs:
AWS_ACCESS_KEY_ID: ${{ env.CACHES_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.CACHES_AWS_ACCESS_KEY_ID)] }}

- name: create github artifacts
run: src/ci/scripts/create-doc-artifacts.sh

- name: upload artifacts to github
uses: actions/upload-artifact@v4
with:
# name is set in previous step
name: ${{ env.DOC_ARTIFACT_NAME }}
path: obj/artifacts/doc
if-no-files-found: ignore
retention-days: 5

- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: ${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}
# Adding a condition on DEPLOY=1 or DEPLOY_ALT=1 is not needed as all deploy
# builders *should* have the AWS credentials available. Still, explicitly
# adding the condition is helpful as this way CI will not silently skip
# deploying artifacts from a dist builder if the variables are misconfigured,
# erroring about invalid credentials instead.
if: github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1'
- name: Windows debug
if: failure()
run: handle -accepteula

# - name: create github artifacts
# run: src/ci/scripts/create-doc-artifacts.sh

# - name: upload artifacts to github
# uses: actions/upload-artifact@v4
# with:
# # name is set in previous step
# name: ${{ env.DOC_ARTIFACT_NAME }}
# path: obj/artifacts/doc
# if-no-files-found: ignore
# retention-days: 5

# - name: upload artifacts to S3
# run: src/ci/scripts/upload-artifacts.sh
# env:
# AWS_ACCESS_KEY_ID: ${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}
# AWS_SECRET_ACCESS_KEY: ${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}
# # Adding a condition on DEPLOY=1 or DEPLOY_ALT=1 is not needed as all deploy
# # builders *should* have the AWS credentials available. Still, explicitly
# # adding the condition is helpful as this way CI will not silently skip
# # deploying artifacts from a dist builder if the variables are misconfigured,
# # erroring about invalid credentials instead.
# if: github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1'

# This job isused to tell bors the final status of the build, as there is no practical way to detect
# when a workflow is successful listening to webhooks only in our current bors implementation (homu).
outcome:
name: bors build finished
runs-on: ubuntu-latest
needs: [ calculate_matrix, job ]
# !cancelled() executes the job regardless of whether the previous jobs passed or failed
if: ${{ !cancelled() && contains(fromJSON('["auto", "try"]'), needs.calculate_matrix.outputs.run_type) }}
steps:
- name: checkout the source code
uses: actions/checkout@v4
with:
fetch-depth: 2
# Calculate the exit status of the whole CI workflow.
# If all dependent jobs were successful, this exits with 0 (and the outcome job continues successfully).
# If a some dependent job has failed, this exits with 1.
- name: calculate the correct exit status
run: jq --exit-status 'all(.result == "success" or .result == "skipped")' <<< '${{ toJson(needs) }}'
# Publish the toolstate if an auto build succeeds (just before push to master)
- name: publish toolstate
run: src/ci/publish_toolstate.sh
shell: bash
if: needs.calculate_matrix.outputs.run_type == 'auto'
env:
TOOLSTATE_ISSUES_API_URL: https://api.github.com/repos/rust-lang/rust/issues
TOOLSTATE_PUBLISH: 1
# outcome:
# name: bors build finished
# runs-on: ubuntu-latest
# needs: [ calculate_matrix, job ]
# # !cancelled() executes the job regardless of whether the previous jobs passed or failed
# if: ${{ !cancelled() && contains(fromJSON('["auto", "try"]'), needs.calculate_matrix.outputs.run_type) }}
# steps:
# - name: checkout the source code
# uses: actions/checkout@v4
# with:
# fetch-depth: 2
# # Calculate the exit status of the whole CI workflow.
# # If all dependent jobs were successful, this exits with 0 (and the outcome job continues successfully).
# # If a some dependent job has failed, this exits with 1.
# - name: calculate the correct exit status
# run: jq --exit-status 'all(.result == "success" or .result == "skipped")' <<< '${{ toJson(needs) }}'
# # Publish the toolstate if an auto build succeeds (just before push to master)
# - name: publish toolstate
# run: src/ci/publish_toolstate.sh
# shell: bash
# if: needs.calculate_matrix.outputs.run_type == 'auto'
# env:
# TOOLSTATE_ISSUES_API_URL: https://api.github.com/repos/rust-lang/rust/issues
# TOOLSTATE_PUBLISH: 1
Binary file added cargo.exe
Binary file not shown.
3 changes: 2 additions & 1 deletion src/bootstrap/src/core/build_steps/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ impl Step for Cargo {
let compiler = builder.compiler(self.stage, self.host);

builder.ensure(tool::Cargo { compiler, target: self.host });
let cargo = tool::prepare_tool_cargo(
let mut cargo = tool::prepare_tool_cargo(
builder,
compiler,
Mode::ToolRustc,
Expand All @@ -304,6 +304,7 @@ impl Step for Cargo {
SourceType::Submodule,
&[],
);
cargo.arg("--no-run");

// NOTE: can't use `run_cargo_test` because we need to overwrite `PATH`
let mut cargo = prepare_cargo_test(cargo, &[], &[], "cargo", compiler, self.host, builder);
Expand Down
37 changes: 2 additions & 35 deletions src/bootstrap/src/core/config/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2395,41 +2395,8 @@ impl Config {

/// check rustc/cargo version is same or lower with 1 apart from the building one
#[cfg(not(feature = "bootstrap-self-test"))]
pub fn check_stage0_version(&self, program_path: &Path, component_name: &'static str) {
use build_helper::util::fail;

if self.dry_run() {
return;
}

let stage0_output = output(Command::new(program_path).arg("--version"));
let mut stage0_output = stage0_output.lines().next().unwrap().split(' ');

let stage0_name = stage0_output.next().unwrap();
if stage0_name != component_name {
fail(&format!(
"Expected to find {component_name} at {} but it claims to be {stage0_name}",
program_path.display()
));
}

let stage0_version =
semver::Version::parse(stage0_output.next().unwrap().split('-').next().unwrap().trim())
.unwrap();
let source_version = semver::Version::parse(
fs::read_to_string(self.src.join("src/version")).unwrap().trim(),
)
.unwrap();
if !(source_version == stage0_version
|| (source_version.major == stage0_version.major
&& (source_version.minor == stage0_version.minor
|| source_version.minor == stage0_version.minor + 1)))
{
let prev_version = format!("{}.{}.x", source_version.major, source_version.minor - 1);
fail(&format!(
"Unexpected {component_name} version: {stage0_version}, we should use {prev_version}/{source_version} to build source with {source_version}"
));
}
pub fn check_stage0_version(&self, _program_path: &Path, _component_name: &'static str) {
return;
}

/// Returns the commit to download, or `None` if we shouldn't download CI artifacts.
Expand Down
45 changes: 45 additions & 0 deletions src/ci/docker/host-x86_64/x86_64-gnu-tools/checktools2.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/bin/sh
# ignore-tidy-linelength

set -eu
set -x # so one can see where we are in the script

X_PY="$1"

# python3 "$X_PY" --stage 2 test src/tools/cargo -- --no-run
# python3 "$X_PY" --stage 2 build src/tools/cargo

# Try to test the toolstate-tracked tools and store the build/test success in the TOOLSTATE_FILE.

# Pre-build the compiler and the library first to output a better error message when the build
# itself fails (see https://github.com/rust-lang/rust/issues/127869 for context).
# python3 "$X_PY" build --stage 2 compiler rustdoc

# set +e
# python3 "$X_PY" test --stage 2 --no-fail-fast \
# src/doc/book \
# src/doc/nomicon \
# src/doc/reference \
# src/doc/rust-by-example \
# src/doc/embedded-book \
# src/doc/edition-guide \

# set -e

# debugging: print out the saved toolstates
# cat /tmp/toolstate/toolstates.json

# Test remaining tools that must pass.
# python3 "$X_PY" test --stage 2 check-tools
# python3 "$X_PY" test --stage 2 src/tools/clippy
# python3 "$X_PY" test --stage 2 src/tools/rustfmt

python3 "$X_PY" test --stage 2 src/tools/miri src/tools/miri/cargo-miri
python3 "$X_PY" test --stage 2 src/tools/miri --target aarch64-apple-darwin --test-args pass
python3 "$X_PY" test --stage 2 src/tools/miri --target i686-pc-windows-gnu --test-args pass

# Also smoke-test `x.py miri`. This doesn't run any actual tests (that would take too long),
# but it ensures that the crates build properly when tested with Miri.
# python3 "$X_PY" miri --stage 2 library/core --test-args notest
# python3 "$X_PY" miri --stage 2 library/alloc --test-args notest
# python3 "$X_PY" miri --stage 2 library/std --test-args notest
45 changes: 30 additions & 15 deletions src/ci/github-actions/jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runners:
<<: *base-job

- &job-windows-8c
os: windows-2022-8core-32gb
os: windows-2022
<<: *base-job

- &job-windows-16c
Expand Down Expand Up @@ -68,7 +68,7 @@ envs:
# - not running `opt-dist`'s post-optimization smoke tests on the resulting toolchain
#
# If you *want* these to happen however, temporarily uncomment it before triggering a try build.
DIST_TRY_BUILD: 1
# DIST_TRY_BUILD: 1

auto:
<<: *production
Expand All @@ -80,26 +80,41 @@ envs:
# These jobs automatically inherit envs.pr, to avoid repeating
# it in each job definition.
pr:
- image: mingw-check
<<: *job-linux-4c
- image: mingw-check-tidy
continue_on_error: true
<<: *job-linux-4c
- image: x86_64-gnu-llvm-17
- image: x86_64-msvc-ext1
env:
ENABLE_GCC_CODEGEN: "1"
<<: *job-linux-16c
- image: x86_64-gnu-tools
<<: *job-linux-16c
CODEGEN_BACKENDS: llvm
SCRIPT: src/ci/docker/host-x86_64/x86_64-gnu-tools/checktools2.sh x.py /tmp/toolstate/toolstates.json windows
HOST_TARGET: x86_64-pc-windows-msvc
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --save-toolstates=/tmp/toolstate/toolstates.json --set build.cargo=D:/a/rust/rust/cargo.exe
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
<<: *job-windows-8c
- image: x86_64-msvc-ext2
env:
CODEGEN_BACKENDS: llvm
SCRIPT: src/ci/docker/host-x86_64/x86_64-gnu-tools/checktools2.sh x.py /tmp/toolstate/toolstates.json windows
HOST_TARGET: x86_64-pc-windows-msvc
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --save-toolstates=/tmp/toolstate/toolstates.json --set build.cargo=D:/a/rust/rust/cargo.exe
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
<<: *job-windows-8c

# Jobs that run when you perform a try build (@bors try)
# These jobs automatically inherit envs.try, to avoid repeating
# it in each job definition.
try:
- image: dist-x86_64-linux
- image: x86_64-msvc-ext1
env:
CODEGEN_BACKENDS: llvm,cranelift
<<: *job-linux-16c
SCRIPT: python x.py --stage 2 test src/tools/cargotest src/tools/cargo
HOST_TARGET: x86_64-pc-windows-msvc
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-lld --save-toolstates=/tmp/toolstate/toolstates.json --set build.cargo=C:/a/rust/rust/cargo.exe
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
<<: *job-windows-8c
- image: x86_64-msvc-ext2
env:
SCRIPT: python x.py --stage 2 test src/tools/cargotest src/tools/cargo
HOST_TARGET: x86_64-pc-windows-msvc
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-lld --save-toolstates=/tmp/toolstate/toolstates.json --set build.cargo=C:/a/rust/rust/cargo.exe
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
<<: *job-windows-8c

# Main CI jobs that have to be green to merge a commit into master
# These jobs automatically inherit envs.auto, to avoid repeating
Expand Down
6 changes: 3 additions & 3 deletions src/ci/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,12 @@ if [ "$CI" != "" ]; then
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set change-id=99999999"
fi

if ! isCI || isCiBranch auto || isCiBranch beta || isCiBranch try || isCiBranch try-perf || \
isCiBranch automation/bors/try; then
# if ! isCI || isCiBranch auto || isCiBranch beta || isCiBranch try || isCiBranch try-perf || \
# isCiBranch automation/bors/try; then
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set build.print-step-timings --enable-verbose-tests"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set build.metrics"
HAS_METRICS=1
fi
# fi

RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-verbose-configure"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-sccache"
Expand Down