Skip to content

Commit

Permalink
Build "min" artifacts on CI (#7315)
Browse files Browse the repository at this point in the history
* riscv64: Extend distance trampolines can jump

Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.

* Build "min" artifacts on CI

This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.

No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.

CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.

While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.

* Fix Windows unzip
  • Loading branch information
alexcrichton authored Oct 23, 2023
1 parent a5d9bbe commit a80da75
Show file tree
Hide file tree
Showing 12 changed files with 295 additions and 106 deletions.
2 changes: 1 addition & 1 deletion .github/actions/binary-compatible-builds/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ if (process.env.CENTOS !== undefined) {
return;
}

const name = process.env.INPUT_NAME;
const name = process.env.INPUT_NAME.replace(/-min$/, '');

child_process.execFileSync('docker', [
'build',
Expand Down
52 changes: 18 additions & 34 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ jobs:
outputs:
run-full: ${{ steps.calculate.outputs.run-full }}
test-matrix: ${{ steps.calculate.outputs.test-matrix }}
build-matrix: ${{ steps.calculate.outputs.build-matrix }}
test-capi: ${{ steps.calculate.outputs.test-capi }}
build-fuzz: ${{ steps.calculate.outputs.build-fuzz }}
audit: ${{ steps.calculate.outputs.audit }}
Expand Down Expand Up @@ -161,6 +162,9 @@ jobs:
echo "test-matrix={\"include\":$(echo $matrix)}" >> $GITHUB_OUTPUT
echo "$matrix"
matrix="$(node ./ci/build-build-matrix.js)"
echo "build-matrix={\"include\":$(echo $matrix)}" >> $GITHUB_OUTPUT
if [ "$run_full" = "true" ]; then
echo run-full=true >> $GITHUB_OUTPUT
echo test-capi=true >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -658,62 +662,42 @@ jobs:
# Perform release builds of `wasmtime` and `libwasmtime.so`. Builds a variety
# of platforms and architectures and then uploads the release artifacts to
# this workflow run's list of artifacts.
#
# Note that the full matrix is computed by `ci/build-build-matrix.js`.
build:
needs: determine
if: needs.determine.outputs.run-full
name: Release build for ${{ matrix.build }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
include:
- build: x86_64-linux
os: ubuntu-latest
- build: x86_64-macos
os: macos-latest
- build: aarch64-macos
os: macos-latest
target: aarch64-apple-darwin
- build: x86_64-windows
os: windows-latest
- build: x86_64-mingw
os: windows-latest
target: x86_64-pc-windows-gnu
- build: aarch64-linux
os: ubuntu-latest
target: aarch64-unknown-linux-gnu
- build: s390x-linux
os: ubuntu-latest
target: s390x-unknown-linux-gnu
- build: riscv64gc-linux
os: ubuntu-latest
target: riscv64gc-unknown-linux-gnu
fail-fast: false
matrix: ${{ fromJson(needs.determine.outputs.build-matrix) }}
steps:
- uses: actions/checkout@v3
with:
submodules: true

- uses: ./.github/actions/install-rust
with:
toolchain: ${{ matrix.rust }}
- run: |
rustup component add rust-src
rustup target add ${{ matrix.target }}
# On one builder produce the source tarball since there's no need to produce
# it everywhere
- run: ./ci/build-src-tarball.sh
if: matrix.build == 'x86_64-linux'
- uses: ./.github/actions/binary-compatible-builds
with:
name: ${{ matrix.build }}
- run: |
echo CARGO_BUILD_TARGET=${{ matrix.target }} >> $GITHUB_ENV
rustup target add ${{ matrix.target }}
if: matrix.target != ''
# Build `wasmtime` and executables. Note that we include some non-default
# features so the # release artifacts can be maximally feature-ful.
- run: $CENTOS cargo build --release --bin wasmtime --features all-arch,component-model

# Build `libwasmtime.so`
- run: $CENTOS cargo build --release --manifest-path crates/c-api/Cargo.toml
- run: $CENTOS ./ci/build-release-artifacts.sh "${{ matrix.build }}" "${{ matrix.target }}"

# Assemble release artifats appropriate for this platform, then upload them
# Assemble release artifacts appropriate for this platform, then upload them
# unconditionally to this workflow's files so we have a copy of them.
- run: ./ci/build-tarballs.sh "${{ matrix.build }}" "${{ matrix.target }}"

- uses: actions/upload-artifact@v3
with:
name: bins-${{ matrix.build }}
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/publish-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
- run: ./ci/merge-artifacts.sh

# Deploy the `gh-pages.tar.gz` artifact to the `gh-pages` branch.
- run: tar xf gh-pages.tar.gz
working-directory: gh-pages
Expand All @@ -41,10 +43,6 @@ jobs:

- run: npm install --production
working-directory: .github/actions/github-release
- run: |
mkdir dist
mv -t dist bins-*/*.tar.*
mv -t dist bins-*/*.{zip,msi,wasm}
- name: Publish Release
uses: ./.github/actions/github-release
with:
Expand Down
68 changes: 68 additions & 0 deletions ci/build-build-matrix.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
// Small script used to calculate the matrix of builds that are going to be
// done if CI decides to do a release build.
//
// This is a separate script primarily to write out all the release
// targets/platforms once and then duplicate them all with a "min" build.

const array = [
{
// The name of the build which shows up in the name of the artifact for
// Wasmtime's github releases.
"build": "x86_64-linux",
// The GitHub Actions platform that this build runs on
"os": "ubuntu-latest",
// The Rust target that will be used for the build.
"target": "x86_64-unknown-linux-gnu",
},
{
"build": "aarch64-linux",
"os": "ubuntu-latest",
"target": "aarch64-unknown-linux-gnu",
},
{
"build": "s390x-linux",
"os": "ubuntu-latest",
"target": "s390x-unknown-linux-gnu",
},
{
"build": "riscv64gc-linux",
"os": "ubuntu-latest",
"target": "riscv64gc-unknown-linux-gnu",
},
{
"build": "x86_64-macos",
"os": "macos-latest",
"target": "x86_64-apple-darwin",
},
{
"build": "aarch64-macos",
"os": "macos-latest",
"target": "aarch64-apple-darwin",
},
{
"build": "x86_64-windows",
"os": "windows-latest",
"target": "x86_64-pc-windows-msvc",
},
{
"build": "x86_64-mingw",
"os": "windows-latest",
"target": "x86_64-pc-windows-gnu",
},
];

const builds = [];
for (let build of array) {
// Perform a "deep clone" roundtripping through JSON for a copy of the build
// that's normal
build.rust = 'stable';
builds.push(JSON.parse(JSON.stringify(build)));

// Next generate a "min" build and add it to the builds list. Min builds
// require Nightly rust due to some nightly build options that are configured.
build.build += '-min';
build.rust = 'nightly-2023-10-10';
builds.push(build);
}

console.log(JSON.stringify(builds));
40 changes: 40 additions & 0 deletions ci/build-release-artifacts.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/bash

# A script to build the release artifacts of Wasmtime into the `target`
# directory. For now this is the CLI and the C API. Note that this script only
# produces the artifacts through Cargo and doesn't package things up. That's
# intended for the `build-tarballs.sh` script.
#
# This script takes a Rust target as its first input and optionally a parameter
# afterwards which can be "-min" to indicate that a minimal build should be
# produced with as many features as possible stripped out.

set -ex

build=$1
target=$2

# Default build flags for release artifacts. Leave debugging for
# builds-from-source which have richer information anyway, and additionally the
# CLI won't benefit from catching unwinds and neither will the C API so use
# panic=abort in both situations.
export CARGO_PROFILE_RELEASE_STRIP=debuginfo
export CARGO_PROFILE_RELEASE_PANIC=abort

if [[ "$build" = *-min ]]; then
# Configure a whole bunch of compile-time options which help reduce the size
# of the binary artifact produced.
export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
export RUSTFLAGS=-Zlocation-detail=none
export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
export CARGO_PROFILE_RELEASE_LTO=true
flags="-Zbuild-std=std,panic_abort --no-default-features -Zbuild-std-features="
flags="$flags --features disable-logging"
else
# For release builds the CLI is built a bit more feature-ful than the Cargo
# defaults to provide artifacts that can do as much as possible.
bin_flags="--features all-arch,component-model"
fi

cargo build --release $flags --target $target -p wasmtime-cli $bin_flags
cargo build --release $flags --target $target -p wasmtime-c-api
2 changes: 2 additions & 0 deletions ci/build-src-tarball.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,5 @@ cargo vendor > .cargo/config.toml
tar -czf /tmp/$pkgname.tar.gz --transform "s/^\./$pkgname/S" --exclude=.git .
mkdir -p dist
mv /tmp/$pkgname.tar.gz dist/

rm .cargo/config.toml
106 changes: 63 additions & 43 deletions ci/build-tarballs.sh
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
#!/bin/bash

# A small script used for assembling release tarballs for both the `wasmtime`
# binary and the C API. This is executed with two arguments, mostly coming from
# the CI matrix.
# binary and the C API. This is executed with two arguments, mostly coming
# from the CI matrix.
#
# * The first argument is the name of the platform, used to name the release
# * The second argument is the "target", if present, currently only for
# cross-compiles
# * The first argument is the name of the "build", used to name the release.
# * The second argument is the Rust target that the build was performed for.
#
# This expects the build to already be done and will assemble release artifacts
# in `dist/`

set -ex

platform=$1
build=$1
target=$2

rm -rf tmp
Expand All @@ -25,8 +24,15 @@ if [[ $GITHUB_REF == refs/heads/release-* ]]; then
tag=v$(./ci/print-current-version.sh)
fi

bin_pkgname=wasmtime-$tag-$platform
api_pkgname=wasmtime-$tag-$platform-c-api
# For *-min builds produce the same named artifacts as the normal build and
# they'll get unioned together in a later step in the CI.
build_pkgname=$build
if [[ $build == *-min ]]; then
build_pkgname=${build%-min}
fi

bin_pkgname=wasmtime-$tag-$build_pkgname
api_pkgname=wasmtime-$tag-$build_pkgname-c-api

mkdir tmp/$api_pkgname
mkdir tmp/$api_pkgname/lib
Expand All @@ -37,47 +43,61 @@ cp LICENSE README.md tmp/$bin_pkgname
cp -r crates/c-api/include tmp/$api_pkgname
cp crates/c-api/wasm-c-api/include/wasm.h tmp/$api_pkgname/include

fmt=tar
if [ "$platform" = "x86_64-windows" ]; then
cp target/release/wasmtime.exe tmp/$bin_pkgname
cp target/release/{wasmtime.dll,wasmtime.lib,wasmtime.dll.lib} tmp/$api_pkgname/lib
fmt=zip

# Generate a `*.msi` installer for Windows as well
export WT_VERSION=`cat Cargo.toml | sed -n 's/^version = "\([^"]*\)".*/\1/p'`
"$WIX/bin/candle" -arch x64 -out target/wasmtime.wixobj ci/wasmtime.wxs
"$WIX/bin/light" -out dist/$bin_pkgname.msi target/wasmtime.wixobj -ext WixUtilExtension
rm dist/$bin_pkgname.wixpdb
elif [ "$platform" = "x86_64-mingw" ]; then
cp target/x86_64-pc-windows-gnu/release/wasmtime.exe tmp/$bin_pkgname
cp target/x86_64-pc-windows-gnu/release/{wasmtime.dll,libwasmtime.a,libwasmtime.dll.a} tmp/$api_pkgname/lib
fmt=zip
elif [ "$platform" = "x86_64-macos" ]; then
# Postprocess the macOS dylib a bit to have a more reasonable `LC_ID_DYLIB`
# directive than the default one that comes out of the linker when typically
# doing `cargo build`. For more info see #984
install_name_tool -id "@rpath/libwasmtime.dylib" target/release/libwasmtime.dylib
cp target/release/wasmtime tmp/$bin_pkgname
cp target/release/libwasmtime.{a,dylib} tmp/$api_pkgname/lib
elif [ "$platform" = "aarch64-macos" ]; then
install_name_tool -id "@rpath/libwasmtime.dylib" target/aarch64-apple-darwin/release/libwasmtime.dylib
cp target/aarch64-apple-darwin/release/wasmtime tmp/$bin_pkgname
cp target/aarch64-apple-darwin/release/libwasmtime.{a,dylib} tmp/$api_pkgname/lib
elif [ "$target" = "" ]; then
cp target/release/wasmtime tmp/$bin_pkgname
cp target/release/libwasmtime.{a,so} tmp/$api_pkgname/lib
else
cp target/$target/release/wasmtime tmp/$bin_pkgname
cp target/$target/release/libwasmtime.{a,so} tmp/$api_pkgname/lib
# For *-min builds rename artifacts with a `-min` suffix to avoid eventual
# clashes with the normal builds when the tarballs are unioned together.
if [[ $build == *-min ]]; then
min="-min"
fi

fmt=tar

case $build in
x86_64-windows*)
cp target/$target/release/wasmtime.exe tmp/$bin_pkgname/wasmtime$min.exe
cp target/$target/release/wasmtime.dll tmp/$api_pkgname/lib/wasmtime$min.dll
cp target/$target/release/wasmtime.lib tmp/$api_pkgname/lib/wasmtime$min.lib
cp target/$target/release/wasmtime.dll.lib tmp/$api_pkgname/lib/wasmtime$min.dll.lib
fmt=zip

if [ "$min" = "" ]; then
# Generate a `*.msi` installer for Windows as well
export WT_VERSION=`cat Cargo.toml | sed -n 's/^version = "\([^"]*\)".*/\1/p'`
"$WIX/bin/candle" -arch x64 -out target/wasmtime.wixobj ci/wasmtime.wxs
"$WIX/bin/light" -out dist/$bin_pkgname.msi target/wasmtime.wixobj -ext WixUtilExtension
rm dist/$bin_pkgname.wixpdb
fi
;;

x86_64-mingw*)
cp target/$target/release/wasmtime.exe tmp/$bin_pkgname/wasmtime$min.exe
cp target/$target/release/wasmtime.dll tmp/$api_pkgname/lib/wasmtime$min.dll
cp target/$target/release/libwasmtime.a tmp/$api_pkgname/lib/libwasmtime$min.a
cp target/$target/release/libwasmtime.dll.a tmp/$api_pkgname/lib/libwasmtime$min.dll.a
fmt=zip
;;

*-macos*)
# Postprocess the macOS dylib a bit to have a more reasonable `LC_ID_DYLIB`
# directive than the default one that comes out of the linker when typically
# doing `cargo build`. For more info see #984
install_name_tool -id "@rpath/libwasmtime$min.dylib" target/$target/release/libwasmtime.dylib
cp target/$target/release/wasmtime tmp/$bin_pkgname/wasmtime$min
cp target/$target/release/libwasmtime.a tmp/$api_pkgname/lib/libwasmtime$min.a
cp target/$target/release/libwasmtime.dylib tmp/$api_pkgname/lib/libwasmtime$min.dylib
;;

*)
cp target/$target/release/wasmtime tmp/$bin_pkgname/wasmtime$min
cp target/$target/release/libwasmtime.a tmp/$api_pkgname/lib/libwasmtime$min.a
cp target/$target/release/libwasmtime.so tmp/$api_pkgname/lib/libwasmtime$min.so
;;
esac


mktarball() {
dir=$1
if [ "$fmt" = "tar" ]; then
# this is a bit wonky, but the goal is to use `xz` with threaded compression
# to ideally get better performance with the `-T0` flag.
tar -cvf - -C tmp $dir | xz -9 -T0 > dist/$dir.tar.xz
tar -czvf dist/$dir.tar.gz -C tmp $dir
else
# Note that this runs on Windows, and it looks like GitHub Actions doesn't
# have a `zip` tool there, so we use something else
Expand Down
Loading

0 comments on commit a80da75

Please sign in to comment.