-
Notifications
You must be signed in to change notification settings - Fork 10.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(ci): improvements to gha workflows #7089
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why propose this
By adopting this re-usable workflows pattern, the build gets much easier to understand and follow, because there is just one workflow driving all the jobs for a given PR or push.
Here is a side-by-side comparison of what it feels like in the UI. Left is before, right is after:
![Screenshot 2024-03-08 at 10 41 25 AM](https://private-user-images.githubusercontent.com/171897/311333978-fa26059f-a511-46dc-8208-df38e8febef7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDYzNzEsIm5iZiI6MTcyNDQwNjA3MSwicGF0aCI6Ii8xNzE4OTcvMzExMzMzOTc4LWZhMjYwNTlmLWE1MTEtNDZkYy04MjA4LWRmMzhlOGZlYmVmNy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwOTQxMTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00YzliYjZmOTBhMzUzNjMxY2UyM2ZkOWFmZGEzYWZjODBmZDJmYWU4N2EyYTQwNWRmMmE0NGY0OWZkOTQ5YTA3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.8lwmx8qbFVotJ5Ge1j1-E8h3-fUDwygA-DrxT0bcvkQ)
The job graph on the Summary page is now accurate:
![Screenshot 2024-03-08 at 10 42 11 AM](https://private-user-images.githubusercontent.com/171897/311334138-04ba56cc-33c2-4074-bd81-cdd324668f9f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDYzNzEsIm5iZiI6MTcyNDQwNjA3MSwicGF0aCI6Ii8xNzE4OTcvMzExMzM0MTM4LTA0YmE1NmNjLTMzYzItNDA3NC1iZDgxLWNkZDMyNDY2OGY5Zi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwOTQxMTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zNDQ2ODNjODQzMDAwZTlkOWVjZDRmMTYxZmMzMmU1ZDFjZjNhZTMyYTEzNzFmMjEwYjk2YjM5MGJhMWJiNjg1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.MzyVgH6tT70-CYlrpuNgbGsy84VS5u0pqLIfxTO4scU)
Dependency Graph
Additionally, this PR helps Guava support the GitHub Dependency Graph. Guava's own dependencies will show up in the GitHub repo, and people who depend on Guava can see it in the graph. By supporting the Dependency Graph, we can also use Dependency Review:
![Screenshot 2024-03-08 at 10 43 04 AM](https://private-user-images.githubusercontent.com/171897/311334361-80080912-b704-41f9-9eb7-c85015b70516.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDYzNzEsIm5iZiI6MTcyNDQwNjA3MSwicGF0aCI6Ii8xNzE4OTcvMzExMzM0MzYxLTgwMDgwOTEyLWI3MDQtNDFmOS05ZWI3LWM4NTAxNWI3MDUxNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwOTQxMTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yNzY1ZDkyZWFiZTczMmJkNDkyNWUxYzkyYzc4MTcxMmU2YjYwYzVmNmE1ZGQwZTE4MWQyOGYyMzdlMzkwODlmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.WLrnflstBamdTh9D9XcyjnxEBxEL5wkkBjYq1dJC7YY)
PRs
Duplicate work was eliminated between the previous push
and pull_request
events. From a PR, things now all gather under one job:
![Screenshot 2024-03-08 at 10 48 56 AM](https://private-user-images.githubusercontent.com/171897/311335582-1af66bc5-2643-4945-8be2-2f634089024d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDYzNzEsIm5iZiI6MTcyNDQwNjA3MSwicGF0aCI6Ii8xNzE4OTcvMzExMzM1NTgyLTFhZjY2YmM1LTI2NDMtNDk0NS04YmUyLTJmNjM0MDg5MDI0ZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwOTQxMTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lYzcyMjA0NmVjYzQwZGYwNDZmZGVmYTE1MWVkZGEzN2UzYjZhZmY5NzE2ZjJmOTMwYWQ3MGNiZjkwZjYxOWEwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.OcGs00A0MWc4HaLKrSLDGyWBo4E30kMORt6xa89OXOQ)
Clicking Details to the right on any job brings you to the same unified summary screen.
Builds, tests, and checks, self-organize because they are all under one job, so they sort hierarchically:
![Screenshot 2024-03-08 at 10 49 32 AM](https://private-user-images.githubusercontent.com/171897/311335701-4c8f058c-da86-44aa-a7d4-c72eaf26f404.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDYzNzEsIm5iZiI6MTcyNDQwNjA3MSwicGF0aCI6Ii8xNzE4OTcvMzExMzM1NzAxLTRjOGYwNThjLWRhODYtNDRhYS1hN2Q0LWM3MmVhZjI2ZjQwNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwOTQxMTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00YzhhNjIwNTg0NTA5MTIwOTk5MTg0Y2FkNGQ0ZmQxNjZlN2VjNmFlZTExNTIxNWI3ZTc4MGRiZjgxYjdjNGZlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.xjuKuieQZcpnuxoZUrLC8qQhh8GLD_Q9qjiaV7B6ePs)
# Guava GitHub CI | ||
# --------------------------------------------------------------------------------------------------------------------- | ||
# This is the main CI build on GitHub for the Google Guava project. This workflow is not invoked directly; instead, the | ||
# `on.pr.yml` and `on.push.yml` workflows kick in on PR and push events, respectively, and call this workflow as a | ||
# Reusable Workflow. | ||
# | ||
# This workflow can be tested independently of the entrypoint flow through the `workflow_dispatch` hook, which adds a | ||
# button within the UI of the GitHub repository. You can trigger the workflow from here: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docs at the top of each workflow.
workflow_call: | ||
inputs: | ||
provenance: | ||
type: boolean | ||
description: "Provenance" | ||
default: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workflow_call
allows reuse of the build and test workflows.
run: | | ||
echo "Building SLSA provenance material..." | ||
ls guava/target/*.jar guava-gwt/target/*.jar guava-testlib/target/*.jar | ||
echo "hashes=$(sha256sum guava/target/*.jar guava-gwt/target/*.jar guava-testlib/target/*.jar | base64 -w0)" >> ./provenance-hashes.txt | ||
cat ./provenance-hashes.txt >> "$GITHUB_OUTPUT" | ||
echo "Gathered provenance hashes:" | ||
cat ./provenance-hashes.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Builds SLSA hashes
- name: Maven Dependency Tree Dependency Submission | ||
continue-on-error: true | ||
uses: advanced-security/maven-dependency-submission-action@bfd2106013da0957cdede0b6c39fb5ca25ae375e # v4.0.2 | ||
- name: 'Dependency Review' | ||
uses: actions/dependency-review-action@9129d7d40b8c12c1ed0f60400d00c92d437adcce # v4.1.3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This workflow covers Github Dependency Graph support; the dependency tree is gathered from Maven, published to Github, and then dependency-review-action
is run, which reviews the dependency graph for licensing violations and vulnerabilities.
Both of these actions come directly from the Github team so they are typically very reliable.
.github/workflows/on.pr.yml
Outdated
## Build the library and provenance material, but don't publish | ||
build: | ||
name: "Build" | ||
uses: ./.github/workflows/ci.build.yml | ||
permissions: | ||
actions: write | ||
contents: write | ||
id-token: write | ||
with: | ||
provenance: true | ||
provenance_publish: false | ||
snapshot: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Entrypoint workflow at on.pr.yml
runs on pull_request
events. Build/test steps are invoked by calling into the shared workflows at ci.build.yml
and ci.test.yml
.
cc / @cpovirk I realize this one is going to look big, but I promise it's worth it if you dig in. Totally happy to close and not merge but I think it will make the build much smoother, and adds some dependency security as well. |
The StepSecurity PR (before cleanup): Here is a sample run: |
4b46092
to
1d1d6dd
Compare
Drops the JDK source steps involved with Javadoc builds, as suggested in #7089. There will be some build warnings related to Javadoc until subsequent PRs are merged. Signed-off-by: Sam Gammon <sam@elide.ventures> Gives up on the "inherting" half of #6790. Fixes #7092 RELNOTES=n/a PiperOrigin-RevId: 614681928
Drops the JDK source steps involved with Javadoc builds, as suggested in #7089. There will be some build warnings related to Javadoc until subsequent PRs are merged. Signed-off-by: Sam Gammon <sam@elide.ventures> Gives up on the "inherting" half of #6790. Fixes #7092 RELNOTES=n/a PiperOrigin-RevId: 614693592
bb90f37
to
85ed7db
Compare
This needs a little bit of cleanup -- mostly an autorebase -- but otherwise it is ready for review @cpovirk |
Okay, excellent, thank you for approving CI workflows by the way. That helps a bunch because I can test running these on an unauthorized fork. Consequently, the two failures there (SLSA Provenance and CodeQL) are happening because the fork (properly) does not have permissions to |
cafe44e
to
56bb0f0
Compare
e3e2cd9
to
0403cf7
Compare
Rebasing this shortly since #7087 was merged |
- chore: apply 'Harden Runner' auditing to all ci tasks - chore: apply `persist-credentials: false` to checkout tasks - chore: publish dependency graph and add dependency review check - chore: add codeql scan job (temp) Bumps [actions/checkout](https://github.com/actions/checkout) from 3.6.0 to 4.1.1. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3.6.0...b4ffde6) Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 2.5.1 to 4.1.3. - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@0efb1d1...9129d7d) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major - dependency-name: actions/dependency-review-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: StepSecurity Bot <bot@stepsecurity.io> Signed-off-by: Sam Gammon <sam@elide.ventures> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0403cf7
to
5bde323
Compare
Provenance/Security stuff
The Sigstore plugin is made by Google and Googlers (the infamous @loosebazooka) and it has worked well in the past purely with a GitHub token in CI. I am happy to test it if desired. ![]() ![]() ![]() Notes on naming: For instance, maybe this should be put in SPDX document sample: Multi-JVM testingSubject to advice on #7087, the build currently maintains the same JVM matrix, and on a subsequent PR I will add multi-JVM testing in one job with Maven toolchains. Any of these can be rolled back and of course this entire PR is optional for the other changes in flight in #7093 and #7094. |
This change refactors the main CI workflow into two new workflows, `on.pr.yml` and `on.push.yml`, which each call into the exiting CI job as a reusable workflow. This has the nice benefit of putting all tests, checks, builds, etc., on one screen during development on GitHub, allows customization of the PR vs. push flow, and yet keeps behavior fully consistent between the two. - chore: move ci jobs to `workflow_call` trigger - chore: add entrypoint jobs for PR and Push events - chore: cleanup permissions and dispatch checks/tests Signed-off-by: Sam Gammon <sam@elide.ventures>
This changeset switches the StepSecurity hardening action to enforced mode, where previously it was running in `audit` mode. Now, audit logs have been gathered and it is time to seal off the list of accessible network endpoints for a given job. - chore: gather and apply network endpoints for each job - chore: move to `block` mode for `egress-policy` in `step-security/harden-runner` Signed-off-by: Sam Gammon <sam@elide.ventures>
This changeset adds SLSA 3+ provenance support to the workflow. The main CI run has now been split into two: `ci.build.yml`, which only builds the library and is provenance-capable, and `ci.test.yml`, which is the previous CI logic. The regular build logic is applied only on push, and can be applied on PRs too, with publish of provenance material turned off. The test suite is invoked from PRs. The workflows have been split into build/test phases to avoid publishing provenance data and GitHub artifacts for build matrix outputs. JARs are uniform across OS targets, so there is no need to gather and publish for more than Ubuntu. - feat: add slsa support to build workflow - chore: split `test` into `build` and `test` workflows - chore: use new workflows (build/test) from push/pr triggers Signed-off-by: Sam Gammon <sam@elide.ventures>
Fails the build if any downloaded dependencies fail their checksum verification. - chore: add `--strict-checksums` flag to `mvnw` calls in ci
- chore: don't rebuild javadoc during tests in ci - chore: don't run with gpg enabled in ci Signed-off-by: Sam Gammon <sam@elide.ventures>
Adds two build parameters - `publishing.repository.snapshots`: Snapshot repo to deploy to - `publishing.repository.releases`: Releases repo to deploy to Both default to their current values, Sonatype. This small inert change allows a fork to easily publish to a different repository without resorting to a code change. Signed-off-by: Sam Gammon <sam@elide.ventures>
This changeset adds the Maven Sigstore plugin for use during publishing to Sonatype and other public repositories. - chore: add sigstore plugin to build Signed-off-by: Sam Gammon <sam@elide.ventures>
Signed-off-by: Sam Gammon <sam@elide.ventures>
Signed-off-by: Sam Gammon <sam@elide.ventures>
I recall a discussion in the OpenSSF SBOM Everywhere on creating a best practice guide on where to place SBOMs, but I couldn't find a link to the document/discussion - @joshbressers may be able to provide a pointer. My personal advice is that the SPDX documents be placed in the directory adjacent to any release artifacts so that it can be picked up by any downstream consumers. If the SPDX document is uploaded to Maven central next to the deployed .jar file, the SPDX maven plugin should recognize it when guava is used in any downstream projects that use the SPDX maven plugin. BTW - Thanks for adding the SPDX documents to the builds! I use guava in some of my projects, and this will improve the fidelity of the SPDX data I produce for those projects. |
73794d5
to
c5846e1
Compare
env: | ||
CI_DEPLOY_USERNAME: ${{ secrets.CI_DEPLOY_USERNAME }} | ||
CI_DEPLOY_PASSWORD: ${{ secrets.CI_DEPLOY_PASSWORD }} | ||
run: ./util/deploy_snapshot.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sgammon Thanks for opening this PR. It would be great to harden the build pipelines for Guava artifacts. I think one issue with the current workflow though is that only the snapshot artifacts are automatically published and not the main artifacts. So, the SLSA provenances can only attest to the snapshot artifacts. We also need to automate the publishing of main artifacts to Maven Central, and generate SLSA provenances for them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@behnazh-w Yes, you're right. I wanted to get things started and churning, and then, with good PR feedback, I am happy to extend the SLSA provenance changes to other parts of the workflow. I believe there are Maven and Bazel generators that could be put to good use.
Just to clarify here, while I am a maintainer of sigstore. It's an openssf/linuxfoundation project. Not a Google project. |
Maven central will accept spdx documents as long as they follow the naming convention, and include the necessary signatures/checksums. I don't have any experience with embedding the sbom doc in the jar. I think maven central is a reasonable place to publish sboms (next to the artifact). Either way I think it's important to make an effort to differentiate dependencies that are shaded in and external dependencies that may or may not change at resolution time. I'm of the opinion that libraries should only include their shaded dependencies in the sbom when publishing to maven central. But I think the jury's still out on this one (@lumjjb, @goneall) |
Dependencies that are included with the distributed artifact(s) can use the |
Gotcha. It sounds like it doesn't make sense to include SBOMs within the JAR itself, so I'll unwind that part and hold it alongside the JAR as an output artifact. |
Summary
Applies updates to GHA CI, and refactors into Reusable Workflows. Hardens supply chain security for Guava both in CI and release. Fixes and closes #7088.
Related PRs
Security hardening
Enclosed features
Changelog
Expand for full changelog
Action updates
Bumps actions/checkout from 3.6.0 to 4.1.1.
Release notes
Sourced from actions/checkout's releases.
Changelog
Sourced from actions/checkout's changelog.
... (truncated)
Commits
b4ffde6
Link to release page from what's new section (#1514)8530928
Correct link to GitHub Docs (#1511)7cdaf2f
Update CODEOWNERS to Launch team (#1510)8ade135
Prepare 4.1.0 release (#1496)c533a0a
Add support for partial checkout filters (#1396)72f2cec
Update README.md for V4 (#1452)3df4ab1
Release 4.0.0 (#1447)8b5e8b7
Support fetching without the --progress option (#1067)97a652b
Update default runtime to node20 (#1436)StepSecurity PR
The PR content below describes the hardening applied by StepSecurity.
Harden Runner
Harden-Runner is an open-source security agent for the GitHub-hosted runner to prevent software supply chain attacks. It prevents exfiltration of credentials, detects tampering of source code during build, and enables running jobs without
sudo
access.Harden runner usage
You can find link to view insights and policy recommendation in the build log
Please refer to documentation to find more details.
Detect Vulnerabilities with SAST Workflow
Static Code Analysis (also known as Source Code Analysis) is usually performed as part of a Code Review (also known as clear-box testing) and is carried out at the Implementation phase of a Security Development Lifecycle (SDL). Static Code Analysis commonly refers to the running of Static Code Analysis tools that attempt to highlight possible vulnerabilities within ‘static’ (non-running) source code by using techniques such as Taint Analysis and Data Flow Analysis.
Add Dependency Review Workflow
The Dependency Review Workflow enforces dependency reviews on your pull requests. The action scans for vulnerable versions of dependencies introduced by package version changes in pull requests, and warns you about the associated security vulnerabilities. This gives you better visibility of what's changing in a pull request, and helps prevent vulnerabilities being added to your repository.