PGO: update the existing benchmarks workflow to enable PGO builds #13884

1pkg · 2024-08-15T00:37:08Z

Motivation/summary

This PR implements changes outlined in #13859. It updates the existing benchmarks workflow to run standalone APM Server instance that produces a relevant CPU profile for PGO, then it copies, uploads and injects the obtained CPU profile into a PR, see example.

Benchmarks

The existing benchmarks results turned to be too unreliable to base PGO on. Because of the underlying dependency on ElasticSearch the difference in the throughput results could go above 10% from a workflow to workflow. The table below provides a view with the existing benchmarks results sample.

This all renders incremental PGO performance gains hard to observe and measure. Therefore, in this PR a new benchmark mode is introduced, which swaps ElasticSearch with a stubbed API http server (Moxy). Thus allowing us to better isolate and elevate APM Server performance component inside the benchmarks. The table below provides a view with the new isolated benchmarks results sample.

Using the benchmarks result sample data we can clearly observe that the results deviation for the new benchmark mode is in an order of magnitude lower in comparison to the existing ES based benchmarks. And now PGO performance improvements could be reliably observed.

The standalone APM Server benchmarks mode consists of running 3 separate EC2 instances in a VPC for apmbench, apm-server and moxy. Existing benchmark_executor and standalone_apm_server terraform modules are reused and a similar new terraform module moxy is created.

Results

PGO enabled builds show 5% performance gain on average across the standalone APM Server benchmarks workflow.

Checklist

Update CHANGELOG.asciidoc
Documentation has been updated

For functional changes, consider:

Is it observable through the addition of either logging or metrics?
Is its use being published in telemetry to enable product improvement?
Have system tests been added to avoid regression?

How to test these changes

To observe and validate the changes please refer to the indexed PGO benchmarks results.

Related issues

#13859

profile.

mergify · 2024-08-15T00:37:53Z

This pull request does not have a backport label. Could you fix it @1pkg? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-7.17 is the label to automatically backport to the 7.17 branch.
backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.

NOTE: backport-skip has been added to this pull request.

1pkg · 2024-10-01T23:13:33Z

Final results after feedback from @v1v to properly set github access token.

The PGO standalone benchmark workflow run link -> resulted in the next PGO update PR link.

While the old benchmark against ES cloud works as expected without regression link.

axw

Nice work - thank you for all the cleanups along the way!

testing/benchmark/variables.tf

testing/benchmark/outputs.tf

testing/benchmark/main.tf

testing/benchmark/Makefile

.github/workflows/benchmarks.yml

axw · 2024-10-02T08:55:52Z

.github/workflows/benchmarks.yml

+      - name: Open PGO PR
+        if: ${{ env.RUN_STANDALONE == 'true' && github.ref == 'refs/heads/main' }}
+        run: make push-pgo-pr
+        env:
+          WORKSPACE_PATH: ${{ github.workspace }}
+          PROFILE_PATH: ${{ env.WORKING_DIRECTORY }}/${{ env.BENCHMARK_CPU_OUT }}
+          GITHUB_TOKEN: ${{ steps.get_token.outputs.token }}
+          WORKFLOW: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/attempts/${{ github.run_attempt }}


I wonder if instead of creating a new PR on every benchmark run, could we just push a commit to the branch?

I think it's slightly risky to enable auto pushes to main branch right away, I'd prefer to start with more controlled PR based approach so we develop the confidence that this pipeline works well. Afterwards we can simplify and enable the direct merge.

I also added a small update to the push-pgo-pr script so it enables auto merging too for PRs. This way we will only need to give it 1 approval and the pipeline tests need to pass.

Fair enough.

axw

LGTM, thank you!

Add a benchmark workflow mode with automation to collect, preserve, and inject CPU profiles, enabling PGO builds. The new workflow will run on a schedule and raise a special pull request that includes the most recent representative CPU profile, which will be inserted as the `default.pgo` file into the main package and automatically used in the build pipeline. The actual schedule and the model for raising pull requests with updated profiles are subject to further revisions. This new workflow mode uses a lightweight output destination - a mock proxy (Moxy) from apm-perf to better isolate the performance component of the APM Server. (cherry picked from commit 5af8cf4)

…) (#14245) Add a benchmark workflow mode with automation to collect, preserve, and inject CPU profiles, enabling PGO builds. The new workflow will run on a schedule and raise a special pull request that includes the most recent representative CPU profile, which will be inserted as the `default.pgo` file into the main package and automatically used in the build pipeline. The actual schedule and the model for raising pull requests with updated profiles are subject to further revisions. This new workflow mode uses a lightweight output destination - a mock proxy (Moxy) from apm-perf to better isolate the performance component of the APM Server. (cherry picked from commit 5af8cf4) Co-authored-by: Kostiantyn Masliuk <1pkg@protonmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Update existing benchmarks workflow to copy, upload and inject PGO

7188d4a

profile.

1pkg self-assigned this Aug 15, 2024

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Aug 15, 2024

Merge branch 'main' into inject-build-pgo-profile

f0b9ecd

1pkg force-pushed the inject-build-pgo-profile branch from c874a6e to f77b98c Compare August 21, 2024 23:47

Only upload benchmarks result from main branch.

39ca00b

1pkg force-pushed the inject-build-pgo-profile branch 2 times, most recently from 576dbb8 to b710c57 Compare August 22, 2024 21:21

Test benchmarks open PGO action.

2ac9c68

1pkg force-pushed the inject-build-pgo-profile branch 7 times, most recently from 1727304 to a6abd96 Compare August 22, 2024 21:56

Test benchmarks workflow add permissions for pull requests.

a320c3d

1pkg force-pushed the inject-build-pgo-profile branch from a6abd96 to a320c3d Compare August 22, 2024 21:58

1pkg added 2 commits August 22, 2024 15:00

Finalize PGO benchmark pipeline update.

a2c42ea

Copy CPU profile to the workspace dir.

cd3c7e1

1pkg force-pushed the inject-build-pgo-profile branch from 8622b95 to cd3c7e1 Compare August 23, 2024 16:12

1pkg added 5 commits August 26, 2024 10:21

Merge branch 'main' into inject-build-pgo-profile

d76d2fd

Put PGO profile into main pkg.

ef3ca41

Use more self-descriptive title and body for PGO PR.

eed7373

Merge branch 'main' into inject-build-pgo-profile

7cd274f

Limit cpu profile size in benchtest.

6bbd47d

1pkg force-pushed the inject-build-pgo-profile branch from 77b0ec8 to 6bbd47d Compare September 10, 2024 04:03

v1v added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 10, 2024

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Sep 10, 2024

1pkg force-pushed the inject-build-pgo-profile branch from 0ed821e to e60e927 Compare October 1, 2024 22:40

1pkg requested a review from v1v October 1, 2024 22:42

delete explicit user git config

d5cfbb9

kruskall previously approved these changes Oct 2, 2024

View reviewed changes

v1v previously approved these changes Oct 2, 2024

View reviewed changes

axw previously approved these changes Oct 2, 2024

View reviewed changes

Merge branch 'main' into inject-build-pgo-profile

13fddd9

1pkg dismissed stale reviews from kruskall, v1v, and axw via 77af1fa October 2, 2024 17:30

1pkg requested review from axw and v1v October 2, 2024 17:47

1pkg force-pushed the inject-build-pgo-profile branch 2 times, most recently from c3f4efd to 0d68b85 Compare October 2, 2024 18:24

address review comments

aae7c2f

1pkg force-pushed the inject-build-pgo-profile branch from 0d68b85 to aae7c2f Compare October 2, 2024 18:39

1pkg requested a review from kruskall October 2, 2024 18:54

axw approved these changes Oct 3, 2024

View reviewed changes

Merge branch 'main' into inject-build-pgo-profile

57ab107

1pkg merged commit 5af8cf4 into main Oct 3, 2024
16 checks passed

1pkg deleted the inject-build-pgo-profile branch October 3, 2024 02:19

mergify bot mentioned this pull request Oct 3, 2024

[8.x] PGO: update the existing benchmarks workflow to enable PGO builds (backport #13884) #14245

Merged

2 tasks

This was referenced Oct 3, 2024

PGO: Move to a simpler process to merge the profiles #14254

Open

PGO: Reduce CPU profile size used for PGO #14255

Closed

PGO: optimize collected profile file size #14256

Merged

mergify bot mentioned this pull request Oct 4, 2024

[8.x] PGO: optimize collected profile file size (backport #14256) #14267

Merged

2 tasks

1pkg mentioned this pull request Oct 16, 2024

changelog: 8.15.3 release #14377

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PGO: update the existing benchmarks workflow to enable PGO builds #13884

PGO: update the existing benchmarks workflow to enable PGO builds #13884

1pkg commented Aug 15, 2024 •

edited

Loading

mergify bot commented Aug 15, 2024

1pkg commented Oct 1, 2024

axw left a comment

axw Oct 2, 2024

1pkg Oct 2, 2024

1pkg Oct 2, 2024

axw Oct 3, 2024

axw left a comment

PGO: update the existing benchmarks workflow to enable PGO builds #13884

PGO: update the existing benchmarks workflow to enable PGO builds #13884

Conversation

1pkg commented Aug 15, 2024 • edited Loading

Motivation/summary

Benchmarks

Results

Checklist

How to test these changes

Related issues

mergify bot commented Aug 15, 2024

1pkg commented Oct 1, 2024

axw left a comment

Choose a reason for hiding this comment

axw Oct 2, 2024

Choose a reason for hiding this comment

1pkg Oct 2, 2024

Choose a reason for hiding this comment

1pkg Oct 2, 2024

Choose a reason for hiding this comment

axw Oct 3, 2024

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

1pkg commented Aug 15, 2024 •

edited

Loading