Upload per-database diagnostic SARIFs on green and red runs #1556

angelapwen · 2023-03-03T02:02:12Z

On failed runs we now call codeql database diagnostic-export to create and upload a SARIF file for per-database diagnostics. If this file was not created, we fall back on the original behavior of uploading the result of codeql export diagnostics.

On successful runs, we add the --sarif-include-diagnostics option to the codeql database interpret-results command.

Both the addition of the option to the interpret-results call, and the codeql database diagnostic-export call, are gated behind the ExportDiagnosticsEnabled feature flag.

We test the green run scenario with an integration test that adds a diagnostic prior to the analyze step, with the feature flag environment variable override to enable the feature. We check the SARIF file generated to make sure it includes the manually added diagnostic.

[GitHub internal only]: we test the red run scenario by writing a similar workflow as the one in the integration test, but failing the workflow after adding the diagnostic. See run and corresponding tool status page for the surfaced diagnostic.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

Move the definition of the variable from util to shared-environment, and rename.

Adds `codeql database export-diagnostics` command and `CODEQL_ACTION_IS_DATABASE_CLUSTER` internal environment variable

src/init-action-post-helper.ts

henrymercer

In the green runs path, we call codeql database interpret-results once per database to produce a SARIF file per DB, combine all these SARIF files together, and then submit a single SARIF file. This has different behaviour to uploading each SARIF file separately, since code scanning will overwrite the existing data for that category each time you upload a new SARIF file.

I think for red runs we can do something simpler — config.dbLocation should always be a DB cluster (at least for all CodeQL versions that support diagnostics). Therefore we should be able to run codeql database export-diagnostics once on that DB cluster and submit the resulting SARIF file.

angelapwen · 2023-03-03T20:26:23Z

Unrelated CI failures that I'll fix in a separate PR!

henrymercer

Looking good so far. An integration test might be tricky to write due to most of the work happening in post-init. We could work around that by creating a separate Action for the test (like we did for the query filters tests), but I'd be happy to merge this with detailed unit tests instead.

src/init-action-post-helper.ts

src/codeql.ts

angelapwen · 2023-03-14T00:02:38Z

I've addressed the existing comments other than testing. I need to update some of the existing unit tests to reflect the new behavior, as well as write new ones to confirm that the appropriate export command is called as expected ✍️

…gnostic

src/init-action-post-helper.ts

henrymercer

Nice!

pr-checks/checks/config-export.yml

pr-checks/checks/diagnostics-export.yml

henrymercer · 2023-03-20T10:55:38Z

pr-checks/checks/diagnostics-export.yml

+        const diagnosticToolExecutionNotification = toolExecutionNotifications[toolExecutionNotifications.length - 1];
+        if (diagnosticToolExecutionNotification.descriptor.id !== 'lang/diagnostics/example' || diagnosticToolExecutionNotification.message.text !== 'Plaintext message') {
+          core.setFailed('`toolExecutionNotification` related to this diagnostic was not found in SARIF');
+        }


I think this will work with the current implementation, but we probably don't want to rely on the ordering of the SARIF file in general. Could we convert this to instead check inclusion?

Yep! Done. Not sure if there's a cleaner way to check for inclusion (maybe if I used a filter and checked if there were any elements left afterwards..?)

Filtering might be slightly neater. You could then also check we have exactly one diagnostic with that ID.

Ah, that's a benefit — done!

pr-checks/checks/diagnostics-export.yml

src/init-action-post-helper.test.ts

src/codeql.ts

Co-authored-by: Henry Mercer <henry.mercer@me.com>

henrymercer

I'd recommend using some functional list methods to simplify the tests, otherwise LGTM

angelapwen added 4 commits March 2, 2023 17:53

Refactor golang autobuild env variable

5d87d01

Move the definition of the variable from util to shared-environment, and rename.

Add database diagnostic command, cluster env var

6b27d47

Adds `codeql database export-diagnostics` command and `CODEQL_ACTION_IS_DATABASE_CLUSTER` internal environment variable

Upload per-database failed SARIFs, if applicable

0277876

Add autogenerated files

b780f5b

angelapwen commented Mar 3, 2023

View reviewed changes

src/init-action-post-helper.ts Outdated Show resolved Hide resolved

angelapwen requested a review from henrymercer March 3, 2023 02:03

henrymercer reviewed Mar 3, 2023

View reviewed changes

Add database export-diagnostics command

1d4190a

angelapwen force-pushed the failed-database-diagnostic branch from e3b8c1f to 1d4190a Compare March 3, 2023 19:48

henrymercer reviewed Mar 6, 2023

View reviewed changes

src/init-action-post-helper.ts Outdated Show resolved Hide resolved

src/init-action-post-helper.ts Outdated Show resolved Hide resolved

src/codeql.ts Show resolved Hide resolved

angelapwen added 4 commits March 8, 2023 13:02

Call export diagnostics if there is no database

e6c90ea

Add new export_diagnostics feature flag

1f1b68b

Pass "--sarif-include-diagnostics" when feature flag on

3200bf1

Resolve merge conflicts

1e4efb1

angelapwen force-pushed the failed-database-diagnostic branch from bb15428 to 1e4efb1 Compare March 13, 2023 22:54

Remove --sarif-include-diagnostics from diagnotics export

ec89543

angelapwen force-pushed the failed-database-diagnostic branch from 0a196c3 to e5a000f Compare March 13, 2023 23:39

Fix export-diagnostics call

6cce51b

angelapwen force-pushed the failed-database-diagnostic branch from e5a000f to 6cce51b Compare March 13, 2023 23:45

angelapwen marked this pull request as ready for review March 14, 2023 00:00

angelapwen requested a review from a team as a code owner March 14, 2023 00:00

angelapwen changed the title ~~[WIP] Upload per-database diagnostic SARIFs on failed runs~~ Upload per-database diagnostic SARIFs on failed runs Mar 14, 2023

angelapwen added 2 commits March 14, 2023 15:38

Improve unit tests

ed73efb

Merge remote-tracking branch 'upstream/main' into failed-database-dia…

c193f61

…gnostic

angelapwen force-pushed the failed-database-diagnostic branch from 090a18c to c193f61 Compare March 15, 2023 16:11

henrymercer reviewed Mar 15, 2023

View reviewed changes

src/init-action-post-helper.ts Outdated Show resolved Hide resolved

Gate database export-diagnostics behind feature flag

30ddec6

angelapwen force-pushed the failed-database-diagnostic branch from d3e5288 to b025421 Compare March 15, 2023 23:33

angelapwen dismissed stale reviews from ghost via 3f5acf0 March 17, 2023 21:36

Check SARIF in test for expected properties

18c0c1a

angelapwen force-pushed the failed-database-diagnostic branch from 3f5acf0 to 18c0c1a Compare March 17, 2023 21:52

angelapwen changed the title ~~Upload per-database diagnostic SARIFs on failed runs~~ Upload per-database diagnostic SARIFs on green and red runs Mar 17, 2023

Rename test for consistency, stop db upload

817a0e5

angelapwen force-pushed the failed-database-diagnostic branch from 016ce45 to 817a0e5 Compare March 18, 2023 00:19

Comment/import cleanup

cb3dd65

angelapwen requested a review from henrymercer March 18, 2023 00:28

henrymercer previously approved these changes Mar 20, 2023

View reviewed changes

angelapwen added 3 commits March 20, 2023 12:12

Test nightly-latest for diagnostics, config export

6ea3eb9

Stop relying on SARIF ordering

9e6750d

Print database diagnostics args if unit test fails

1954584

angelapwen dismissed stale reviews from ghost and henrymercer via 1954584 March 20, 2023 19:26

angelapwen and others added 2 commits March 20, 2023 12:29

Apply suggestions from code review

9116cc1

Co-authored-by: Henry Mercer <henry.mercer@me.com>

Add JSDoc comment

0271b54

angelapwen force-pushed the failed-database-diagnostic branch from 92c75df to 0271b54 Compare March 20, 2023 19:33

Improve failure message, some autogenerated fixes

3465159

angelapwen requested a review from henrymercer March 20, 2023 19:37

henrymercer previously approved these changes Mar 20, 2023

View reviewed changes

Refactor inclusion check

e7e7280

angelapwen dismissed henrymercer’s stale review via e7e7280 March 20, 2023 20:11

angelapwen requested a review from henrymercer March 20, 2023 20:12

henrymercer approved these changes Mar 20, 2023

View reviewed changes

angelapwen enabled auto-merge (squash) March 20, 2023 20:23

angelapwen merged commit 3cbd063 into github:main Mar 20, 2023

angelapwen deleted the failed-database-diagnostic branch March 21, 2023 18:12

github-actions bot mentioned this pull request Mar 22, 2023

Merge main into releases/v2 #1601

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload per-database diagnostic SARIFs on green and red runs #1556

Upload per-database diagnostic SARIFs on green and red runs #1556

angelapwen commented Mar 3, 2023 •

edited

Loading

henrymercer left a comment

angelapwen commented Mar 3, 2023

henrymercer left a comment

angelapwen commented Mar 14, 2023

henrymercer left a comment

henrymercer Mar 20, 2023

angelapwen Mar 20, 2023

henrymercer Mar 20, 2023

angelapwen Mar 20, 2023

henrymercer left a comment

Upload per-database diagnostic SARIFs on green and red runs #1556

Upload per-database diagnostic SARIFs on green and red runs #1556

Conversation

angelapwen commented Mar 3, 2023 • edited Loading

Merge / deployment checklist

henrymercer left a comment

Choose a reason for hiding this comment

angelapwen commented Mar 3, 2023

henrymercer left a comment

Choose a reason for hiding this comment

angelapwen commented Mar 14, 2023

henrymercer left a comment

Choose a reason for hiding this comment

henrymercer Mar 20, 2023

Choose a reason for hiding this comment

angelapwen Mar 20, 2023

Choose a reason for hiding this comment

henrymercer Mar 20, 2023

Choose a reason for hiding this comment

angelapwen Mar 20, 2023

Choose a reason for hiding this comment

henrymercer left a comment

Choose a reason for hiding this comment

angelapwen commented Mar 3, 2023 •

edited

Loading