Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RAM and threads options to init action #738

Merged
merged 1 commit into from
Oct 28, 2021
Merged

Add RAM and threads options to init action #738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/__extractor-ram-threads.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 39 additions & 0 deletions .github/workflows/pr-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,3 +393,42 @@ jobs:
# Deliberately don't use TEST_MODE here. This is specifically testing
# the compatibility with the API.
runner/dist/codeql-runner-linux upload --sarif-file src/testdata/empty-sarif.sarif --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

runner-extractor-ram-threads-options:
name: Runner ubuntu extractor RAM and threads options
needs: [check-js, check-node-modules]
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Run init
run: |
runner/dist/codeql-runner-linux init --ram=230 --threads=1 --repository $GITHUB_REPOSITORY --languages java --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd slightly prefer this to use something other than 1 for the threads as 1 could very easily be a default in which case we wouldn't really be checking anything. Maybe 3 as that seems very unlikely to ever be the default value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a decision that I struggled with.

The main challenge here is that the number of threads used defaults to (and is capped at) the number of CPU cores available on the machine. GitHub action runners have 2 CPUs, so both the default and the cap would be 2. Passing any value larger than 1 means that the test will break if it runs on a system with different number of CPUs.

I would love to hear your thoughts. For example, maybe we could assume that the runner will always have at least 2 CPUs and have two tests with threads set to 1 and 2?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, this is a good point. As I mentioned below, I'm actually not sure that capping this is a good idea, but for now let's leave this test as is 🙂


- name: Assert Results
shell: bash
run: |
. ./codeql-runner/codeql-env.sh
if [ "${CODEQL_RAM}" != "230" ]; then
echo "CODEQL_RAM is '${CODEQL_RAM}' instead of 230"
exit 1
fi
if [ "${CODEQL_EXTRACTOR_JAVA_RAM}" != "230" ]; then
echo "CODEQL_EXTRACTOR_JAVA_RAM is '${CODEQL_EXTRACTOR_JAVA_RAM}' instead of 230"
exit 1
fi
if [ "${CODEQL_THREADS}" != "1" ]; then
echo "CODEQL_THREADS is '${CODEQL_THREADS}' instead of 1"
exit 1
fi
if [ "${CODEQL_EXTRACTOR_JAVA_THREADS}" != "1" ]; then
echo "CODEQL_EXTRACTOR_JAVA_THREADS is '${CODEQL_EXTRACTOR_JAVA_THREADS}' instead of 1"
exit 1
fi
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## [UNRELEASED]

No user facing changes.
- The `init` step of the Action now supports `ram` and `threads` inputs to limit resource use of CodeQL extractors. These inputs also serve as defaults to the subsequent `analyze` step, which finalizes the database and executes queries. [#738](https://github.com/github/codeql-action/pull/738)

## 1.0.21 - 28 Oct 2021

Expand Down
14 changes: 12 additions & 2 deletions analyze/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@ inputs:
required: false
default: "brutal"
ram:
description: Override the amount of memory in MB to be used by CodeQL. By default, almost all the memory of the machine is used.
description: >-
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the input descriptions here are going to vary between the two actions, we should add an exception to the custom query that's producing the code scanning warning on this file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I edited queries/inconsistent-action-input.ql to exclude inputs named ram or threads from the query.

The amount of memory in MB that can be used by CodeQL for database finalization and query execution.
By default, this action will use the same amount of memory as previously set in the "init" action.
adityasharad marked this conversation as resolved.
Show resolved Hide resolved
If the "init" action also does not have an explicit "ram" input, this action will use most of the
memory available in the system (which for GitHub-hosted runners is 6GB for Linux, 5.5GB for Windows,
and 13GB for macOS).
required: false
add-snippets:
description: Specify whether or not to add code snippets to the output sarif file.
Expand All @@ -29,7 +34,12 @@ inputs:
required: false
default: "false"
threads:
description: The number of threads to be used by CodeQL.
description: >-
The number of threads that can be used by CodeQL for database finalization and query execution.
By default, this action will use the same number of threads as previously set in the "init" action.
adityasharad marked this conversation as resolved.
Show resolved Hide resolved
If the "init" action also does not have an explicit "threads" input, this action will use all the
hardware threads available in the system (which for GitHub-hosted runners is 2 for Linux and Windows
and 3 for macOS).
required: false
checkout_path:
description: "The path at which the analyzed repository was checked out. Used to relativize any absolute paths in the uploaded SARIF file."
Expand Down
14 changes: 14 additions & 0 deletions init/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ inputs:
source-root:
description: Path of the root source code directory, relative to $GITHUB_WORKSPACE.
required: false
ram:
description: >-
The amount of memory in MB that can be used by CodeQL extractors.
By default, CodeQL extractors will use most of the memory available in the system
(which for GitHub-hosted runners is 6GB for Linux, 5.5GB for Windows, and 13GB for macOS).
This input also sets the amount of memory that can later be used by the "analyze" action.
required: false
threads:
description: >-
The number of threads that can be used by CodeQL extractors.
By default, CodeQL extractors will use all the hardware threads available in the system
(which for GitHub-hosted runners is 2 for Linux and Windows and 3 for macOS).
This input also sets the number of threads that can later be used by the "analyze" action.
required: false
outputs:
codeql-path:
description: The path of the CodeQL binary used for analysis
Expand Down
75 changes: 75 additions & 0 deletions lib/analyze-action-env.test.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions lib/analyze-action-env.test.js.map
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

75 changes: 75 additions & 0 deletions lib/analyze-action-input.test.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions lib/analyze-action-input.test.js.map
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.