Improve reliability and performance when using the `autobuild` build mode #2235

henrymercer · 2024-04-12T14:11:45Z

This PR directly traces autobuild scripts when a build mode is specified to the init Action.

Before, the init Action did not know whether the workflow would call the autobuilder or run manual build scripts, so we always had to start indirect tracing in case we needed to trace manual build steps.

However in new-style workflows where the build mode is specified, this is no longer necessary, and we only need to use indirect tracing when the build mode is manual. For the autobuild build mode, we can avoid starting indirect tracing and directly trace the autobuild script using database trace-command. We expect this to provide reliability and performance benefits, which we will validate.

This is behind the autobuild_direct_tracing_enabled feature flag.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

Behind a flag, for now

aeisenberg

Looks great.

aeisenberg · 2024-04-12T19:17:26Z

CHANGELOG.md

@@ -16,6 +16,7 @@ Note that the only difference between `v2` and `v3` of the CodeQL Action is the
  We recommend removing any references to these from your workflows. For more information, see the release notes for CodeQL Action v3.23.0 and v2.23.0.
 - Automatically overwrite an existing database if found on the filesystem. [#2229](https://github.com/github/codeql-action/pull/2229)
 - Bump the minimum CodeQL bundle version to 2.12.6. [#2232](https://github.com/github/codeql-action/pull/2232)
+- We are rolling out a feature in April 2024 that improves the reliability and performance of analyzing code when using the `autobuild` build mode. This feature uses direct tracing to trace the build when the `autobuild` build mode is specified using the `build-mode` input to the `init` Action. [#2335](https://github.com/github/codeql-action/pull/2335)


There's a lot going on in this changelog. Can you add some links to external docs around direct tracing. Also, is there a github changelog post for the build-mode changes you can link to?

Fair point — I'll leave the details out of the changelog note, and link some docs about build modes.

aeisenberg · 2024-04-12T19:26:17Z

pr-checks/checks/autobuild-direct-tracing.yml

@@ -0,0 +1,30 @@
+name: "Autobuild direct tracing"
+description: "An end-to-end integration test of a Java repository built using 'build-mode: autobuild', with direct tracing enabled"
+operatingSystems: ["ubuntu"]


Should we be testing on windows, too?

Yes, I think this makes sense. It doesn't really give us any additional confidence that the code changes are correct, but it does give us better confidence that direct tracing is working as expected.

aeisenberg · 2024-04-12T19:28:27Z

pr-checks/checks/autobuild-direct-tracing.yml

+name: "Autobuild direct tracing"
+description: "An end-to-end integration test of a Java repository built using 'build-mode: autobuild', with direct tracing enabled"
+operatingSystems: ["ubuntu"]
+versions: ["latest", "nightly-latest"]


Not something to fix in this PR, but how can we ensure that as we release new versions of the CLI, we add them to the matrix? There are other integration tests that only run on "latest", "nightly-latest" because when we wrote them, those were the only versions that supported the feature being tested.

Could we add a minimum-version property so that tests that specify it run on all CLI versions >= the minimum version?

I like this idea, but I think it's beyond the scope of this PR. I'll file an issue.

angelapwen · 2024-04-12T21:46:05Z

src/codeql.ts

+      if (config.buildMode === BuildMode.Autobuild) {
+        applyAutobuildAzurePipelinesTimeoutFix();
+      }


I'm not sure I understand why're now applying this fix only if we're using autobuild/direct tracing 🤔 it looks like we set those options unconditionally in the past.

I think this PR should be additive when it comes to applying this fix — previously we applied it to the autobuild Action only, now we also apply it to database trace-command calls.

angelapwen · 2024-04-15T21:06:12Z

CHANGELOG.md

@@ -16,7 +16,7 @@ Note that the only difference between `v2` and `v3` of the CodeQL Action is the
  We recommend removing any references to these from your workflows. For more information, see the release notes for CodeQL Action v3.23.0 and v2.23.0.
 - Automatically overwrite an existing database if found on the filesystem. [#2229](https://github.com/github/codeql-action/pull/2229)
 - Bump the minimum CodeQL bundle version to 2.12.6. [#2232](https://github.com/github/codeql-action/pull/2232)
- We are rolling out a feature in April 2024 that improves the reliability and performance of analyzing code when using the `autobuild` build mode. This feature uses direct tracing to trace the build when the `autobuild` build mode is specified using the `build-mode` input to the `init` Action. [#2335](https://github.com/github/codeql-action/pull/2335)
+- We are rolling out a feature in April/May 2024 that improves the reliability and performance of analyzing code when analyzing a compiled language with the `autobuild` [build mode](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes). [#2335](https://github.com/github/codeql-action/pull/2335)


Just noticed PR number should be 2235 instead of 2335 😸

henrymercer added 6 commits April 12, 2024 16:37

Move BuildMode to util.ts

5b74166

Use direct tracing when autobuild build mode specified

e37d0f3

Behind a flag, for now

Use extraction via build mode in autobuild Action when available

3d49faa

Apply network timeout fix to extraction with direct tracing

8f057a3

Run autobuild script via direct tracing when feature enabled

cbe29f5

Add changelog note

2eaad47

henrymercer force-pushed the henrymercer/autobuild-with-direct-tracing branch from 8859bb4 to 2eaad47 Compare April 12, 2024 16:07

henrymercer changed the base branch from main to henrymercer/feature-flags-with-tool-feature-dependencies April 12, 2024 16:07

Add PR check for direct tracing

aa4a08d

henrymercer marked this pull request as ready for review April 12, 2024 17:00

henrymercer requested a review from a team as a code owner April 12, 2024 17:00

Base automatically changed from henrymercer/feature-flags-with-tool-feature-dependencies to main April 12, 2024 17:00

Merge branch 'main' into henrymercer/autobuild-with-direct-tracing

f21d2d2

henrymercer changed the title ~~Improve the reliability and performance of analyzing code when using the autobuild build mode~~ Improve reliability and performance when using the autobuild build mode Apr 12, 2024

aeisenberg reviewed Apr 12, 2024

View reviewed changes

angelapwen reviewed Apr 12, 2024

View reviewed changes

henrymercer self-assigned this Apr 15, 2024

henrymercer added 3 commits April 15, 2024 18:26

Run PR check on Windows too

59771dd

Improve changelog note

14eab91

Merge branch 'main' into henrymercer/autobuild-with-direct-tracing

ea78e51

henrymercer requested review from aeisenberg and angelapwen April 15, 2024 17:51

angelapwen previously approved these changes Apr 15, 2024

View reviewed changes

Fix PR link in changelog

db2b739

henrymercer dismissed angelapwen’s stale review via db2b739 April 16, 2024 11:34

henrymercer requested a review from angelapwen April 16, 2024 11:34

angelapwen approved these changes Apr 16, 2024

View reviewed changes

henrymercer merged commit 18111b6 into main Apr 16, 2024
320 checks passed

henrymercer deleted the henrymercer/autobuild-with-direct-tracing branch April 16, 2024 17:59

github-actions bot mentioned this pull request Apr 17, 2024

Merge main into releases/v3 #2248

Merged

8 tasks

github-actions bot mentioned this pull request Apr 17, 2024

Merge releases/v3 into releases/v2 #2250

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve reliability and performance when using the `autobuild` build mode #2235

Improve reliability and performance when using the `autobuild` build mode #2235

henrymercer commented Apr 12, 2024 •

edited

Loading

aeisenberg left a comment

aeisenberg Apr 12, 2024

henrymercer Apr 15, 2024

aeisenberg Apr 12, 2024

henrymercer Apr 15, 2024

aeisenberg Apr 12, 2024

henrymercer Apr 15, 2024

angelapwen Apr 12, 2024

henrymercer Apr 15, 2024

angelapwen Apr 15, 2024

Improve reliability and performance when using the autobuild build mode #2235

Improve reliability and performance when using the autobuild build mode #2235

Conversation

henrymercer commented Apr 12, 2024 • edited Loading

Merge / deployment checklist

aeisenberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Improve reliability and performance when using the `autobuild` build mode #2235

Improve reliability and performance when using the `autobuild` build mode #2235

henrymercer commented Apr 12, 2024 •

edited

Loading