Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: build relases on gh-actions & support linux arm64 #2629

Merged
merged 6 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: Build & Test

on:
push:
branches: ["*"]

env:
# Cross-compilation for aarch64 requires a different linker
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: aarch64-linux-gnu-gcc

permissions:
contents: read

jobs:
Tests:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
target:
- x86_64-unknown-linux-gnu
- aarch64-unknown-linux-gnu
- x86_64-pc-windows-msvc
- x86_64-apple-darwin
- aarch64-apple-darwin
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to run tests on different archs for the same OS?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHubs Action matrix is a little counter-intuitive. A job for each target (currently 5) and each rustup_toolchain (currently 1) is created. The include part sets the os. We use the native os for each target, except linux arm. I thought testing on the native arch is best when ever possible - but I know very litte about the rust build standards.

If you think testing with cross compiling is better or also needed required let me know the exact test matrix.

rustup_toolchain: [stable]
include:
- os: windows-2022
target: x86_64-pc-windows-msvc
- os: ubuntu-20.04
target: x86_64-unknown-linux-gnu
- os: ubuntu-20.04
target: aarch64-unknown-linux-gnu
- os: macos-13
target: x86_64-apple-darwin
- os: macos-14
target: aarch64-apple-darwin
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install Rust (UNIX)
run: |
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain ${{ matrix.rustup_toolchain }}
echo "##vso[task.setvariable variable=PATH;]$PATH:$HOME/.cargo/bin"
# We need to crosscompile for arm64 linux for now
if [[ "${{ matrix.target }}" == "aarch64-unknown-linux-gnu" ]]; then
echo "Adding aarch64-unknown-linux-gnu target"
sudo apt-get update -y
sudo apt-get install -y make g++ libssl-dev gcc-aarch64-linux-gnu
rustup target add aarch64-unknown-linux-gnu
else
echo "Compiling for ${{matrix.target}}"
fi
if: ${{ ! startsWith(matrix.os, 'windows') }}

- name: Install Rust (Windows)
run: |
curl -sSf -o rustup-init.exe https://win.rustup.rs
./rustup-init.exe -y --default-toolchain ${{ matrix.rustup_toolchain }} --default-host x86_64-pc-windows-msvc
echo "##vso[task.setvariable variable=PATH;]%PATH%;%USERPROFILE%\.cargo\bin"
if: ${{ startsWith(matrix.os, 'windows') }}
hegerdes marked this conversation as resolved.
Show resolved Hide resolved

- name: Cargo build (Native TLS)
run: cargo build --all --no-default-features --features=native-tls && cargo clean

- name: Cargo build (Rust TLS)
run: cargo build --all

- name: Cargo test
run: cargo test --all

- name: Cargo fmt
run: cargo fmt --check
50 changes: 0 additions & 50 deletions .github/workflows/cd-workflow.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
env:
BUILD_DIR: docs/
BUILD_ONLY: true

build_and_deploy:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/master'
Expand Down
173 changes: 173 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
name: Release

on:
push:
tags: ["v*.*.*"]

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
# Cross-compilation for aarch64 requires a different linker
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: aarch64-linux-gnu-gcc

permissions:
contents: read

jobs:
Release-Build:
runs-on: ${{ matrix.os }}
permissions:
contents: read
attestations: write
id-token: write
strategy:
matrix:
target:
- x86_64-unknown-linux-gnu
- aarch64-unknown-linux-gnu
- x86_64-pc-windows-msvc
- x86_64-apple-darwin
- aarch64-apple-darwin
rustup_toolchain: [stable]
include:
- os: windows-2022
target: x86_64-pc-windows-msvc
- os: ubuntu-20.04
target: x86_64-unknown-linux-gnu
- os: ubuntu-20.04
target: aarch64-unknown-linux-gnu
- os: macos-13
target: x86_64-apple-darwin
- os: macos-14
target: aarch64-apple-darwin
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install Rust (UNIX)
if: ${{ ! startsWith(matrix.os, 'windows') }}
run: |
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain ${{ matrix.rustup_toolchain }}
echo "##vso[task.setvariable variable=PATH;]$PATH:$HOME/.cargo/bin"
# We need to crosscompile for arm64 linux for now
if [[ "${{ matrix.target }}" == "aarch64-unknown-linux-gnu" ]]; then
echo "Adding aarch64-unknown-linux-gnu target"
sudo apt-get update -y
sudo apt-get install -y make g++ libssl-dev gcc-aarch64-linux-gnu
rustup target add aarch64-unknown-linux-gnu
else
echo "Compiling for ${{matrix.target}}"
fi

- name: Install Rust (Windows)
if: ${{ startsWith(matrix.os, 'windows') }}
run: |
curl -sSf -o rustup-init.exe https://win.rustup.rs
./rustup-init.exe -y --default-toolchain ${{ matrix.rustup_toolchain }} --default-host ${{ matrix.target }}
echo "##vso[task.setvariable variable=PATH;]%PATH%;%USERPROFILE%\.cargo\bin"
hegerdes marked this conversation as resolved.
Show resolved Hide resolved

- name: Cargo build
run: cargo build --release --target ${{ matrix.target }}

- name: Archive (UNIX)
run: |
mkdir -p artifacts
cp -av target/${{ matrix.target }}/release/zola .
tar -czf ${{ github.event.repository.name }}-${{ github.ref_name }}-${{ matrix.target }}.tar.gz zola
if: ${{ ! startsWith(matrix.os, 'windows') }}

- name: Archive (Windows)
run: |
mkdir -p artifacts
cp target/${{ matrix.target }}/release/zola.exe .
tar -czf ${{ github.event.repository.name }}-${{ github.ref_name }}-${{ matrix.target }}.zip zola.exe
if: ${{ startsWith(matrix.os, 'windows') }}

- name: Attest Build Provenance
uses: actions/attest-build-provenance@v1
continue-on-error: true
with:
subject-path: ${{ github.event.repository.name }}-${{ github.ref_name }}-${{ matrix.target }}.*

- uses: actions/upload-artifact@v4
with:
name: ${{ github.event.repository.name }}-${{ github.ref_name }}-${{ matrix.target }}
path: ${{ github.event.repository.name }}-${{ github.ref_name }}-${{ matrix.target }}.*
if-no-files-found: error
retention-days: 7

Release:
needs: [Release-Build]
runs-on: ubuntu-latest
permissions:
contents: write

steps:
- name: Ensure artifacts dir exists
run: mkdir -p artifacts

- name: Download Artifact
uses: actions/download-artifact@v4
with:
path: artifacts
merge-multiple: true

- name: Release
uses: softprops/action-gh-release@c062e08bd532815e2082a85e87e3ef29c3e6d191
with:
name: ${{ github.ref_name }}
tag_name: ${{ github.ref_name }}
generate_release_notes: true
fail_on_unmatched_files: true
body: |
Welcome to this new release of Zola ${{ github.ref_name }}!

All artifacts are signed with this repos identity using Sigstore.
You can verify the signatures using the `GitHub` CLI.

```shell
gh attestation verify --owner ${{ github.repository_owner }} <my-artifact>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now Azure generates a list of commits since the last release (https://github.com/getzola/zola/releases/tag/v0.19.2), is it possible to keep it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it should generate release notes with generate_release_notes: true. My demo release didn't have any real changes since it was the first release and I'm not working with PRs for active development and debugging.

I'm also using this action here: https://github.com/hegerdes/joplin-plugin-remote-note-pull/releases/tag/v1.3.0 and Pulumi also uses uses with some more changes than my small project, see https://github.com/pulumi/pulumi-awsx/releases

I think they might look a litte different to azures notes though.

Feel fee to add whatever you want to that release body you are the owner, it was just an example for now.

```
token: ${{ secrets.GITHUB_TOKEN }}
prerelease: ${{ contains(github.ref, '-pre') }}
files: artifacts/*
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason behind splitting the Release-Build and Release jobs? Why not do it in 1 job?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build for each target run all on their own isolated runner. So the targets can be build native and in parallel. The runners don't share any storage so wee need to get all build assets on one runner. This is done via upload/download artifacts. The artifacts assets are temporary and will auto delete in one week.

When each runner creates a release we can have race conditions or partly failed release jobs. This ensures the release build for every target worked and only then one release is created. If one target fails -> no release.

The argument prerelease would allow you to create pre-releases when the tag looks like v0.1.0-pre.0 It will not be marked as latest.


Release-Container-Image:
needs: [Release]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up QEMU
uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf

- name: Setup Docker buildx
uses: docker/setup-buildx-action@988b5a0280414f521da01fcc63a27aeeb4b104db

- name: Log in to the Container registry
uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
flavor: latest=false

- name: Build and push Docker image
uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
with:
context: .
push: true
build-args: USE_GH_RELEASE=true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
18 changes: 12 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
FROM rust:slim-bookworm AS builder

RUN apt-get update -y && \
apt-get install -y make g++ libssl-dev && \
rustup target add x86_64-unknown-linux-gnu
ARG USE_GH_RELEASE=false
RUN apt-get update -y && apt-get install -y make g++ libssl-dev curl jq tar gzip

WORKDIR /app
COPY . .

RUN cargo build --release --target x86_64-unknown-linux-gnu

RUN if [ "${USE_GH_RELEASE}" = "true" ]; then \
mkdir -p target/$(uname -m)-unknown-linux-gnu/release && \
export ZOLA_VERSION=$(curl -sL https://api.github.com/repos/getzola/zola/releases/latest | jq -r .name) && \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not reproducible, can you pass the version to the Dockerfile instead?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally agree. Just wanted to check if this approach is fine before going all the way.
I would pass an extra arg like ZOLA_VERSION with. I would set the default to latest and override in CI to the current tag. To enforce reproducibility we could also just fail if USE_GH_RELEASE is set but ZOLA_VERSION is not. Any takes on that?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good

curl -sL --fail --output zola.tar.gz https://github.com/getzola/zola/releases/download/${ZOLA_VERSION}/zola-${ZOLA_VERSION}-$(uname -m)-unknown-linux-gnu.tar.gz && \
tar -xzvf zola.tar.gz zola; \
else \
cargo build --release && \
cp target/$(uname -m)-unknown-linux-gnu/release/zola zola; \
fi && ./zola --version

FROM gcr.io/distroless/cc-debian12
COPY --from=builder /app/target/x86_64-unknown-linux-gnu/release/zola /bin/zola
COPY --from=builder /app/zola /bin/zola
ENTRYPOINT [ "/bin/zola" ]