Skip to content

Commit

Permalink
refactor(ci): update graal release infrastructure
Browse files Browse the repository at this point in the history 
This was a bit of a rabbit hole, but I just went with it. With the
changes to sbt-ci-release, _somehow_ the graal native image generation
was failing on windows. In order to combat that this pr makes some
changes to CI and to the args we're using with Graal. I'll outline the
changes below.

- Migrate to `graalvm/setup-graalvm` for the graal jobs. This is
  necessary if we want to use the newer graal versions. When I tried
  with `setup-scala` I got jabba erros that what I was looking for
  doesn't exist. Plus `setup-scala` isn't _really_ worked on anymore, so
  migrating away is preferable. Plus, when using `setup-graalvm` it's
  easy to get things like the `native-image` command on all platforms.
- Bump to 22.3.0. I tried to just bump slightly, but there is a bug in
  22.1.0 on windows that I hit on oracle/graal#4502.
  It's fixed in the newer ones, so I just bumped up.
- Use `actions/setup-java` instead of `setup-scala` for jvm tests. Same
  reason as above with maintenance, but also built-in sbt cache. For now
  we're still doing a lot on 11 (where the graal stuff is on 17) but I'm
  trying to not change this too much for now until I get everything
  green and releasing. Then I'll address that.
- Move `publish-binaries-windows` into `publish-binaries`. Now that we
  are using `setup-graalvm` it's trivial to just keep these together,
  using `sbt` and just call it a day. No need for two separate ones.
- Remove the windows-specific `graalVMNativeImageCommand`. Again, now
  that we're using `setup-graalvm` we easily have the `native-image`
  command on the PATH so we don't need to worry about all this extra
  stuff.
- Change a few flags:
  - `--no-server` wasn't valid it seemed as we were getting a message on
    every run that it wasn't recognized and ignored.
  - `-H:EnableUrlProtocols` no just uses `--enable-url-protocols`.
    Apparently it's frowned upon to use the `-H` stuff since it's
    internal, so this was just a bit of cleaning up. We do have a couple
    others, but there doesn't seem to be alternatives so I left them as
    is.
  - `--allow-incomlete-classpath` This is the default now, so no need to
    have it.

You can see a green build for all 3 os's [here in my
fork](https://github.com/ckipp01/bloop/actions/runs/3787821939/jobs/6439973910).
But let's see how all the other jobs go.
  • Loading branch information
@ckipp01
ckipp01 committed Dec 27, 2022
1 parent cfc902d commit fa3f197
Show file tree
Hide file tree
Showing 2 changed files with 71 additions and 97 deletions.
156 changes: 69 additions & 87 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,47 +14,56 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: olafurpg/setup-scala@v13
- uses: graalvm/setup-graalvm@v1
with:
java-version: graalvm-ce-java11@21.1.0
version: '22.3.0'
java-version: '17'
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Check formatting
run: ./bin/scalafmt --test

integrations:
name: Build integrations tests
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macOS-latest]
jdk: [graalvm-ce-java11@21.1.0]
steps:
- uses: actions/checkout@v3
with:
submodules: true
- uses: olafurpg/setup-scala@v13
- uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.jdk }}
version: '22.3.0'
java-version: '11'
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Tests
run: |
./bin/sbt-ci.sh \
"sbtBloop/publishLocal" \
"sbtBloop/scripted"
shell: bash

bridges:
name: Test platform bridges
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
jdk: [graalvm-ce-java11@21.1.0]
steps:
- uses: actions/checkout@v2
with:
submodules: true
- uses: olafurpg/setup-scala@v13
- uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.jdk }}
version: '22.3.0'
java-version: '11'
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Tests
run: |
./bin/sbt-ci.sh \
Expand All @@ -71,18 +80,20 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
jdk: [graalvm-ce-java11@21.1.0]
steps:
- uses: actions/checkout@v2
with:
submodules: true
- uses: olafurpg/setup-scala@v13
- uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.jdk }}
version: '22.3.0'
java-version: '11'
github-token: ${{ secrets.GITHUB_TOKEN }}
components: 'native-image'

- name: Tests
run: |
echo $JAVA_HOME
which gu && gu install native-image
./bin/sbt-ci.sh "install" "launcherTest/test"
shell: bash

Expand All @@ -92,18 +103,31 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
jdk: [graalvm-ce-java11@21.1.0, adopt@1.11]
jdk: ["22.3.0", "11"]

name: Test ${{ matrix.os }} -- ${{ matrix.jdk }}
env:
HYDRA_LICENSE: ${{ secrets.HYDRA_LICENSE }}

steps:
- uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0
- uses: olafurpg/setup-scala@v13

- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ matrix.java }}
cache: 'sbt'
if: matrix.jdk == '11'

- uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.jdk }}
version: ${{ matrix.jdk }}
java-version: '11'
if: matrix.jdk == '22.3.0'

- uses: actions/setup-node@v3
with:
node-version: "10.x"
Expand All @@ -114,6 +138,7 @@ jobs:
java -version
[[ $HYDRA_LICENSE == floating-key=* ]] && mkdir -p $HOME/.triplequote && echo "$HYDRA_LICENSE" > "$HOME/.triplequote/hydra.license" || echo "Hydra license file was not created"
shell: bash

- name: Compile and test main projects
run: |
./bin/sbt-ci.sh \
Expand All @@ -125,6 +150,7 @@ jobs:
"frontend/runMain bloop.util.CommandsDocGenerator --test" \
"frontend/runMain bloop.util.CommandsDocGenerator --out ../docs/cli/reference.md"
shell: bash

- name: Check docs are up-to-date
run: |
./bin/check-good-practices.sh
Expand All @@ -137,51 +163,12 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-20.04, macOS-latest]
jdk: [graalvm-ce-java11@21.1.0]
os: [ubuntu-20.04, macOS-latest, windows-latest]
include:
- os: ubuntu-20.04
artifact: bloop-linux
- os: macos-latest
artifact: bloop-macos
env:
CI: true
SKIP_TEST_RESOURCES_GENERATION: true
GITHUB_DEPLOY_KEY: ${{ secrets.GITHUB_DEPLOY_KEY }}
BLOOPOID_GITHUB_TOKEN: ${{ secrets.BLOOPOID_GITHUB_TOKEN }}
ARTIFACT_NAME: ${{ matrix.artifact }}
steps:
- uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0
- uses: olafurpg/setup-scala@v13
with:
java-version: ${{ matrix.jdk }}
- name: Publish GraalVM Native artifacts
run: |
echo $JAVA_HOME
which gu
gu install native-image
./bin/sbt-ci.sh "bloopgun/graalvm-native-image:packageBin"
mkdir bloop-artifacts
cp bloopgun/target/bloopgun-2.12/graalvm-native-image/bloopgun-core bloop-artifacts/$ARTIFACT_NAME
shell: bash
- uses: actions/upload-artifact@v3
with:
name: bloop-artifacts
path: bloop-artifacts/${{ matrix.artifact }}

publish-binaries-windows:
name: Publish binaries for ${{ matrix.os }}
runs-on: ${{ matrix.os }}
if: github.event_name == 'push'
strategy:
fail-fast: false
matrix:
os: [windows-latest]
jdk: [graalvm-ce-java11@21.1.0]
include:
- os: windows-latest
artifact: bloop-windows
env:
Expand All @@ -191,43 +178,38 @@ jobs:
BLOOPOID_GITHUB_TOKEN: ${{ secrets.BLOOPOID_GITHUB_TOKEN }}
ARTIFACT_NAME: ${{ matrix.artifact }}
steps:
- name: Configure Pagefile
uses: al-cheb/configure-pagefile-action@v1.3
- name: Refresh Pagefile
run: |
(Get-CimInstance Win32_PageFileUsage).AllocatedBaseSize
- uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0
- uses: olafurpg/setup-scala@v13

- uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.jdk }}
- name: Install GraalVM Native Image
run: |
echo $JAVA_HOME
"$JAVA_HOME\bin\gu.cmd" install native-image
shell: bash
- name: Pre-load sbt with bash
# FIXME: This step shouldn't be necessary, but sbt fails to find
# org.scala-lang.modules#scala-xml_2.12;1.2.0!scala-xml_2.12.jar when launched with
# cmd.
# Keep the sbt version in sync with `sbt-ci-release.bat`.
run: |
sbt -sbt-version 1.8.0 version
shell: bash
version: '22.3.0'
java-version: '17'
components: 'native-image'
native-image-job-reports: 'true'
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Publish GraalVM Native artifacts
run: >-
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" &&
"%CD%"\bin\sbt-ci-release.bat
shell: cmd
- name: Prepare generated Windows GraalVM binary
run: sbt "bloopgun213/graalvm-native-image:packageBin"

- name: Copy artifacts (windows)
if: matrix.os == 'windows-latest'
run: |
echo $ARTIFACT_NAME
ls bloopgun/target/bloopgun-2.12/graalvm-native-image/bloopgun-core.exe
mkdir bloop-artifacts
cp bloopgun/target/bloopgun-2.12/graalvm-native-image/bloopgun-core.exe bloop-artifacts/$ARTIFACT_NAME
ls bloopgun/target/bloopgun-2.13/graalvm-native-image/bloopgun-core.exe
mkdir -p bloop-artifacts
cp bloopgun/target/bloopgun-2.13/graalvm-native-image/bloopgun-core.exe bloop-artifacts/$ARTIFACT_NAME
shell: bash

- name: Copy artifacts (not windows)
if: matrix.os != 'windows-latest'
run: |
echo $ARTIFACT_NAME
mkdir -p bloop-artifacts
cp bloopgun/target/bloopgun-2.13/graalvm-native-image/bloopgun-core bloop-artifacts/$ARTIFACT_NAME
- uses: actions/upload-artifact@v3
with:
name: bloop-artifacts
Expand All @@ -237,14 +219,13 @@ jobs:
name: Release version on ${{ matrix.os }}
needs:
- publish-binaries
- publish-binaries-windows
runs-on: ${{ matrix.os }}
if: github.event_name == 'push'
strategy:
fail-fast: true
matrix:
os: [ubuntu-20.04]
jdk: [adopt@1.8]

env:
CI: true
SKIP_TEST_RESOURCES_GENERATION: true
Expand All @@ -261,9 +242,10 @@ jobs:
with:
submodules: true
fetch-depth: 0
- uses: olafurpg/setup-scala@v13
- uses: actions/setup-java@v3
with:
java-version: ${{ matrix.jdk }}
distribution: 'temurin'
java-version: '11'
- uses: actions/setup-node@v3
with:
node-version: "10.x"
Expand Down
12 changes: 2 additions & 10 deletions build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -225,11 +225,6 @@ lazy val bloopgunSettings = Seq(
Dependencies.jsoniterMacros % Provided
),
(GraalVMNativeImage / mainClass) := Some("bloop.bloopgun.Bloopgun"),
graalVMNativeImageCommand := {
val oldPath = graalVMNativeImageCommand.value
if (!scala.util.Properties.isWin) oldPath
else "C:/Users/runneradmin/.jabba/jdk/graalvm-ce-java11@21.1.0/bin/native-image.cmd"
},
graalVMNativeImageOptions ++= {
val reflectionFile = (Compile / Keys.sourceDirectory).value./("graal")./("reflection.json")
val securityOverridesFile =
Expand All @@ -240,14 +235,12 @@ lazy val bloopgunSettings = Seq(
s"${securityOverridesFile.getAbsolutePath()} doesn't exist"
)
List(
"--no-server",
"--enable-http",
"--enable-https",
"-H:EnableURLProtocols=http,https",
"--enable-url-protocols=http,https",
"--enable-all-security-services",
"--no-fallback",
s"-H:ReflectionConfigurationFiles=$reflectionFile",
"--allow-incomplete-classpath",
"-H:+ReportExceptionStackTraces",
s"-J-Djava.security.properties=$securityOverridesFile",
s"-Djava.security.properties=$securityOverridesFile",
Expand Down Expand Up @@ -492,8 +485,7 @@ addCommandAlias(
"install",
Seq(
"publishLocal",
// Don't generate graalvm image if running in Windows
if (isWindows) "" else "bloopgun/graalvm-native-image:packageBin",
"bloopgun/graalvm-native-image:packageBin",
s"${frontend.id}/test:compile",
"createLocalHomebrewFormula",
"createLocalScoopFormula",
Expand Down

0 comments on commit fa3f197

Please sign in to comment.