Skip to content

Commit

Permalink
chore(benchmarks): Rehauling Benchmark Workloads (#2716)
Browse files Browse the repository at this point in the history
## Things changes

* using new action plugin:
[benchmark-action/github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark)
* previous library was
[bobheadxi/gobenchdata](https://github.com/bobheadxi/gobenchdata)
* creating alerts when benchmarks results are worse

## Things unchanged

* Pushing benchmarks results onto a local branch, then `benchmark`
repository will fetch from this into his own `gh-pages`
  * direct push is possible but requires PAT token to be used 

## Things TODO (before merging)

* restore benchmarks actions which were manually disabled directly in
the `Action` menu in Github
* drop and recreate branch  `gh-repository`
* fix benchmark tests in Gno repo (see #2711 and #2714)
* Evaluate alternative configuration possibilities
[here](https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file#action-inputs)

## Things to consider (performance)

* Should solve #2432 
* are resources available in the Github basic runners enough for
benchmarks?
* are multiple runs spawned in the same hw conditions?
* consider using [Larger
Runner](https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/managing-larger-runners#adding-a-larger-runner-to-an-organization)
or restoring a [Self-Hosted
Runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-an-organization)
  • Loading branch information
sw360cab authored Sep 26, 2024
1 parent ca9eb4b commit 577c462
Show file tree
Hide file tree
Showing 5 changed files with 48 additions and 56 deletions.
34 changes: 0 additions & 34 deletions .benchmarks/README.md

This file was deleted.

9 changes: 0 additions & 9 deletions .benchmarks/gobenchdata-checks.yml

This file was deleted.

3 changes: 2 additions & 1 deletion .github/workflows/benchmark-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ jobs:
secrets: inherit
with:
publish: false
test-flags: "-short"
test-flags: "-short"
external-data-json-path: "./cache/benchmark-data.json"
1 change: 1 addition & 0 deletions .github/workflows/benchmark-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
workflow_dispatch:
schedule:
- cron: '0 0 * * *' # run on default branch every day

jobs:
publish:
uses: ./.github/workflows/benchmark_template.yml
Expand Down
57 changes: 45 additions & 12 deletions .github/workflows/benchmark_template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,29 +8,62 @@ on:
test-flags:
required: true
type: string
external-data-json-path:
type: string

permissions:
# deployments permission to deploy GitHub pages website
deployments: write
# contents permission to update benchmark contents in gh-pages branch
contents: write

env:
CGO_ENABLED: 0

jobs:
benchmarks:
if: ${{ github.repository == 'gnolang/gno' }}
runs-on: [self-hosted, Linux, X64, benchmark-v1]
runs-on: [self-hosted, Linux, X64, benchmarks]
steps:
- name: checkout
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 1

- uses: actions/setup-go@v5
with:
go-version: "1.22.x"
- name: "gobenchdata publish: ${{ inputs.publish }}"
run: go run go.bobheadxi.dev/gobenchdata@v1 action
env:
INPUT_PRUNE_COUNT: 30
INPUT_GO_TEST_FLAGS: "${{ inputs.test-flags }} -run=^$ -cpu 1,2" # executing only using one and two CPUs to not be dependant on the machine cores.
INPUT_PUBLISH: ${{ inputs.publish }}
INPUT_PUBLISH_BRANCH: gh-benchmarks
INPUT_BENCHMARKS_OUT: benchmarks.json
INPUT_CHECKS: ${{ !inputs.publish }}
INPUT_CHECKS_CONFIG: .benchmarks/gobenchdata-checks.yml

- name: Run benchmark
run: |
go test -benchmem -bench=. ./... -run=^$ \
-cpu 1,2 ${{ inputs.test-flags }} | tee benchmarks.txt
- name: Download previous benchmark data
uses: actions/cache@v4
with:
path: ./cache
key: ${{ runner.os }}-benchmark

- name: Store benchmark results into `gh-benchmarks` branch
uses: benchmark-action/github-action-benchmark@v1
# see https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file#action-inputs
with:
name: Go Benchmarks
tool: 'go'
output-file-path: benchmarks.txt
# Where the previous data file is stored
external-data-json-path: ${{ inputs.external-data-json-path }}
# Show alert with commit comment on detecting possible performance regression
alert-threshold: '20%'
fail-on-alert: true
comment-on-alert: true
# Enable Job Summary for PRs
summary-always: true
github-token: ${{ secrets.GITHUB_TOKEN }}
# NOTE you need to use a separate GITHUB PAT token that has a write access to the specified repository.
# gh-repository: 'github.com/gnolang/benchmarks' # on gh-pages branch
gh-pages-branch: gh-benchmarks
benchmark-data-dir-path: .
auto-push: ${{ inputs.publish }}
alert-comment-cc-users: '@ajnavarro,@thehowl,@zivkovicmilos'

0 comments on commit 577c462

Please sign in to comment.