Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving CodeQL into builds, and making the CodeQL job a re-usable flow #27484

Merged
merged 2 commits into from
Jun 26, 2023
Merged

Moving CodeQL into builds, and making the CodeQL job a re-usable flow #27484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8446412
Building re-usable builds for codeQL
woody-apple Jun 26, 2023
888d5ed
Merge branch 'master' into codeql-reuse
woody-apple Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 111 additions & 15 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,12 @@ on:
pull_request:
merge_group:
workflow_dispatch:

workflow_call:
inputs:
run-codeql:
required: false
type: boolean

concurrency:
group: ${{ github.ref }}-${{ github.workflow }}-${{ (github.event_name == 'pull_request' && github.event.number) || (github.event_name == 'workflow_dispatch' && github.run_number) || github.sha }}
cancel-in-progress: true
Expand Down Expand Up @@ -94,6 +99,11 @@ jobs:
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log

- name: Initialize CodeQL
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/init@v2
with:
languages: "cpp"
- name: Setup Build
run: scripts/build/gn_gen.sh --args="chip_config_memory_debug_checks=true chip_config_memory_debug_dmalloc=false"
- name: Run Build
Expand Down Expand Up @@ -133,6 +143,36 @@ jobs:
path: /tmp/cores/
# Cores are big; don't hold on to them too long.
retention-days: 5
- name: Perform CodeQL Analysis
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/analyze@v2
with:
category: "/language:cpp"
upload: False
output: sarif-results
- name: filter-sarif
if: ${{ inputs.run-codeql }}
uses: advanced-security/filter-sarif@v1
with:
patterns: |
-**/third_party/**
-**/scripts/**
input: "sarif-results/cpp.sarif"
output: "sarif-results/cpp.sarif"

- name: Upload SARIF
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: "sarif-results/cpp.sarif"

- name: Upload loc as a Build Artifact
if: ${{ inputs.run-codeql }}
uses: actions/upload-artifact@v2.2.0
with:
name: sarif-results
path: sarif-results
retention-days: 1
# OBJDIR on linux is > 10K files and takes more than 50 minutes to upload, usually
# having the job timeout.
#
Expand Down Expand Up @@ -178,11 +218,6 @@ jobs:
token: ${{ github.token }}
attempt_limit: 3
attempt_delay: 2000
# - name: Initialize CodeQL
# if: ${{ github.event_name == 'push' && github.event.ref == 'refs/heads/master' }}
# uses: github/codeql-action/init@v1
# with:
# languages: "cpp"
- name: Checkout submodules
run: scripts/checkout_submodules.py --allow-changing-global-git-config --shallow --platform linux
- name: Try to ensure the directories for core dumping exist and we
Expand Down Expand Up @@ -215,7 +250,11 @@ jobs:
path: |
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log

- name: Initialize CodeQL
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/init@v2
with:
languages: "cpp"
- name: Setup and Build Simulated Device
timeout-minutes: 20
run: |
Expand Down Expand Up @@ -302,6 +341,36 @@ jobs:
run: |
./scripts/run_in_build_env.sh \
"./scripts/build/build_examples.py --target linux-fake-tests build"
- name: Perform CodeQL Analysis
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/analyze@v2
with:
category: "/language:cpp"
upload: False
output: sarif-results
- name: filter-sarif
if: ${{ inputs.run-codeql }}
uses: advanced-security/filter-sarif@v1
with:
patterns: |
-**/third_party/**
-**/scripts/**
input: "sarif-results/cpp.sarif"
output: "sarif-results/cpp.sarif"

- name: Upload SARIF
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: "sarif-results/cpp.sarif"

- name: Upload loc as a Build Artifact
if: ${{ inputs.run-codeql }}
uses: actions/upload-artifact@v2.2.0
with:
name: sarif-results
path: sarif-results
retention-days: 1
- name: Uploading core files
uses: actions/upload-artifact@v3
if: ${{ failure() && !env.ACT }}
Expand Down Expand Up @@ -411,11 +480,6 @@ jobs:
token: ${{ github.token }}
attempt_limit: 3
attempt_delay: 2000
#- name: Initialize CodeQL
# if: ${{ github.event_name == 'push' && github.event.ref == 'refs/heads/master' }}
# uses: github/codeql-action/init@v1
# with:
# languages: "cpp"
- name: Checkout submodules
run: scripts/checkout_submodules.py --allow-changing-global-git-config --shallow --platform darwin
- name: Try to ensure the directory for diagnostic log collection exists
Expand Down Expand Up @@ -447,6 +511,11 @@ jobs:
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log

- name: Initialize CodeQL
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/init@v2
with:
languages: "cpp"
- name: Setup and Build Simulated Device
timeout-minutes: 20
run: |
Expand Down Expand Up @@ -495,9 +564,36 @@ jobs:
with:
name: crash-log-darwin
path: ~/Library/Logs/DiagnosticReports/
# - name: Perform CodeQL Analysis
# if: ${{ github.event_name == 'push' && github.event.ref == 'refs/heads/master' }}
# uses: github/codeql-action/analyze@v1
- name: Perform CodeQL Analysis
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/analyze@v2
with:
category: "/language:cpp"
upload: False
output: sarif-results
- name: filter-sarif
if: ${{ inputs.run-codeql }}
uses: advanced-security/filter-sarif@v1
with:
patterns: |
-**/third_party/**
-**/scripts/**
input: "sarif-results/cpp.sarif"
output: "sarif-results/cpp.sarif"

- name: Upload SARIF
if: ${{ inputs.run-codeql }}
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: "sarif-results/cpp.sarif"

- name: Upload loc as a Build Artifact
if: ${{ inputs.run-codeql }}
uses: actions/upload-artifact@v2.2.0
with:
name: sarif-results
path: sarif-results
retention-days: 1
# TODO Log Upload https://github.com/project-chip/connectedhomeip/issues/2227
# TODO https://github.com/project-chip/connectedhomeip/issues/1512

Expand Down
113 changes: 3 additions & 110 deletions .github/workflows/codeql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,116 +25,9 @@ concurrency:
group: ${{ github.ref }}-${{ github.workflow }}-${{ (github.event_name == 'pull_request' && github.event.number) || (github.event_name == 'workflow_dispatch' && github.run_number) || github.sha }}
cancel-in-progress: true

env:
CHIP_NO_LOG_TIMESTAMPS: true
# XXX: Workaround for https://github.com/actions/cache/issues/1141
SEGMENT_DOWNLOAD_TIMEOUT_MINS: 3

jobs:
analyze:
name: CodeQL Analysis
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
permissions:
actions: read
contents: read
security-events: write

container:
image: connectedhomeip/chip-build:0.7.3
volumes:
- "/tmp/log_output:/tmp/test_logs"
options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
net.ipv4.conf.all.forwarding=1 net.ipv6.conf.all.forwarding=1"

strategy:
fail-fast: false
matrix:
language: [ 'cpp' ]
# language: [ 'cpp', 'java', 'javascript', 'python' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby', 'swift' ]
# Use only 'java' to analyze code written in Java, Kotlin or both
# Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support


steps:
- name: Dump GitHub context
env:
GITHUB_CONTEXT: ${{ toJSON(github) }}
run: echo "$GITHUB_CONTEXT"
- name: Dump Concurrency context
env:
CONCURRENCY_CONTEXT: ${{ github.ref }}-${{ github.workflow }}-${{ (github.event_name == 'pull_request' && github.event.number) || (github.event_name == 'workflow_dispatch' && github.run_number) || github.sha }}
run: echo "$CONCURRENCY_CONTEXT"
- uses: Wandalen/wretry.action@v1.3.0
name: Checkout
with:
action: actions/checkout@v3.5.2
with: |
token: ${{ github.token }}
attempt_limit: 3
attempt_delay: 2000
- name: Checkout submodules
run: scripts/checkout_submodules.py --allow-changing-global-git-config --shallow --platform linux
- name: Try to ensure the directories for core dumping exist and we
can write them.
run: |
mkdir /tmp/cores || true
sysctl -w kernel.core_pattern=/tmp/cores/core.%u.%p.%t || true

- uses: Wandalen/wretry.action@v1.3.0
name: Bootstrap cache
continue-on-error: true
timeout-minutes: 10
with:
action: buildjet/cache@v3
attempt_limit: 3
attempt_delay: 2000
with: |
key: ${{ runner.os }}-env-${{ hashFiles('scripts/setup/*', 'third_party/pigweed/**') }}
path: |
.environment
build_overrides/pigweed_environment.gni
- name: Bootstrap
run: bash scripts/bootstrap.sh

- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}

- name: Setup Build
run: scripts/build/gn_gen.sh --args="chip_config_memory_debug_checks=true chip_config_memory_debug_dmalloc=false"
- name: Run Build
run: scripts/run_in_build_env.sh "ninja -C ./out"
- name: Run Tests
run: scripts/tests/gn_tests.sh
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
with:
category: "/language:${{matrix.language}}"
upload: False
output: sarif-results

- name: filter-sarif
uses: advanced-security/filter-sarif@v1
with:
patterns: |
-**/app/tests/**
-**/third_party/**
-**/scripts/**
input: "sarif-results/${{matrix.language}}.sarif"
output: "sarif-results/${{matrix.language}}.sarif"

- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: "sarif-results/${{matrix.language}}.sarif"
uses: project-chip/connectedhomeip/.github/workflows/build.yml@main
with:
run-codeql: true

- name: Upload loc as a Build Artifact
uses: actions/upload-artifact@v2.2.0
with:
name: sarif-results
path: sarif-results
retention-days: 1