Add some dependencies for uploading artifacts #538

edoardopirovano · 2021-06-01T13:25:15Z

This PR adds two dependencies that will be used for future steps that upload artifacts. In particular, the two dependencies added are:

"@actions/artifact": "^0.5.1",
"glob": "^7.1.7",

The first is needed to upload artifacts. The second isn't strictly needed but makes it much easier to select whole folders with many files (such as CodeQL databases) for upload.

This PR also does an update of our dependencies since it seems reasonable to do this while we're touching the package.json.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

aeisenberg

What version of npm are you using? I see that you downgraded the lock file from v2 to v1. And that has changed just about every file. Also, not sure why all of the lib/*.js files were changed, too. Maybe this implicitly changed the version of typescript?

So, can you recreate this change using node v14?

Also, if it's not already in the dev notes or contributing guide, could you add a mention that node >=14 is required?

aeisenberg · 2021-06-01T15:50:19Z

.github/workflows/script/check-node-modules.sh

@@ -14,7 +14,7 @@ npm run removeNPMAbsolutePaths
 # Check that repo is still clean
 if [ ! -z "$(git status --porcelain)" ]; then
    # If we get a fail here then the PR needs attention
-    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' to update"
+    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' on a Mac OS X machine to update"


Suggested change

>&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' on a Mac OS X machine to update"

>&2 echo "Failed: node_modules are not up to date. Run 'npm ci && npm run removeNPMAbsolutePaths' on a Mac OS X machine to update"

Does this need to be on a macos X machine? Technically, BigSur is not X But more importantly, what if you're on linux or windows?s?

Unfortunately, if the command is run on Linux or Windows rather than macOS, the dependency fsevents (which is Mac-specific) is not installed causing test failues (and likely also issues in production on Mac machines). I agree we should have a way of updating dependencies not on macOS but I'm not entirely sure how we can actually resolve the issue. npm isn't really designed for our workflow of checking in dependencies, so as far as I can tell there's no way to make it install a dependency that isn't compatible with the OS you're on.

Oh...that sucks. It's not even possible to explicitly add the fsevents dependency in the package.json?

Sadly not, if we explicitly add the dependency in package.json then doing npm install on Linux doesn't even work:

npm ERR! notsup Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin"} (current: {"os":"linux","arch":"x64"})

edoardopirovano · 2021-06-01T16:54:53Z

Aha, good point. I actually did have an up to date node but had an outdated npm which was using the old lockfile format. I've updated that and sure enough the changes are somewhat more manageable now. I've updated the docs to say we want node 14+ and npm 7+.

src/toolrunner-error-catcher.test.ts

aeisenberg · 2021-06-01T17:11:56Z

.github/workflows/script/check-node-modules.sh

@@ -14,7 +14,7 @@ npm run removeNPMAbsolutePaths
 # Check that repo is still clean
 if [ ! -z "$(git status --porcelain)" ]; then
    # If we get a fail here then the PR needs attention
-    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' to update"
+    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci && npm run removeNPMAbsolutePaths' on a macOS machine to update. Note it is important this command is run on macOS and not any other operating system as there is one dependency (fsevents) that is needed for macOS and may not be installed if the command is run on a Windows or Linux machine."


Just want clarification that this command only works on macos. This is surprising because the workflows themselves successfully run npm ci && npm run removeNPMAbsolutePaths.

Locally for me (Ubuntu 20.04, node 14.17.0 and npm 7.15.1), running npm ci && npm run removeNPMAbsolutePaths removes the fsevents package:

edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps ±  git reset --hard origin/db-upload-deps HEAD is now at 5b7c3a46 Add some dependencies for uploading artifacts edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps  npm ci && npm run removeNPMAbsolutePaths added 564 packages, and audited 565 packages in 8s 56 packages are looking for funding run `npm fund` for details found 0 vulnerabilities > codeql@1.0.0 removeNPMAbsolutePaths > removeNPMAbsolutePaths . --force edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps ±  git status On branch db-upload-deps Your branch is up-to-date with 'origin/db-upload-deps'. Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: node_modules/.package-lock.json deleted: node_modules/fsevents/LICENSE deleted: node_modules/fsevents/README.md deleted: node_modules/fsevents/fsevents.d.ts deleted: node_modules/fsevents/fsevents.js deleted: node_modules/fsevents/fsevents.node deleted: node_modules/fsevents/package.json no changes added to commit (use "git add" and/or "git commit -a")

I agree this is strange, because the PR check that runs the same step remotely also runs on Ubuntu 20.04 and doesn't fail. I even tried invoking the PR check script locally and this still fails:

edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps ±  git reset --hard origin/db-upload-deps HEAD is now at 5b7c3a46 Add some dependencies for uploading artifacts edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps  .github/workflows/script/check-node-modules.sh added 564 packages, and audited 565 packages in 8s 56 packages are looking for funding run `npm fund` for details found 0 vulnerabilities > codeql@1.0.0 removeNPMAbsolutePaths > removeNPMAbsolutePaths . --force Failed: node_modules are not up to date. Run 'npm ci && npm run removeNPMAbsolutePaths' on a macOS machine to update. Note it is important this command is run on macOS and not any other operating system as there is one dependency (fsevents) that is needed for macOS and may not be installed if the command is run on a Windows or Linux machine. On branch db-upload-deps Your branch is up-to-date with 'origin/db-upload-deps'. Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: node_modules/.package-lock.json deleted: node_modules/fsevents/LICENSE deleted: node_modules/fsevents/README.md deleted: node_modules/fsevents/fsevents.d.ts deleted: node_modules/fsevents/fsevents.js deleted: node_modules/fsevents/fsevents.node deleted: node_modules/fsevents/package.json no changes added to commit (use "git add" and/or "git commit -a") ✘ edoardo@EdoardoDesktop  ~/codeql-action   db-upload-deps ± 

I'm completely stumped on why the PR check would work but locally it does not. I still think advising to only update dependencies on Mac OS is reasonable, since if the command is done there it always seems to work.

Hmmm...tried this on my windows vm and I'm seeing the same behaviour. I think we can punt this for now. It's documented at least. Thanks for digging into this.

aeisenberg · 2021-06-01T22:51:52Z

.github/workflows/script/check-node-modules.sh

@@ -14,7 +14,7 @@ npm run removeNPMAbsolutePaths
 # Check that repo is still clean
 if [ ! -z "$(git status --porcelain)" ]; then
    # If we get a fail here then the PR needs attention
-    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' to update"
+    >&2 echo "Failed: node_modules are not up to date. Run 'npm ci && npm run removeNPMAbsolutePaths' on a macOS machine to update. Note it is important this command is run on macOS and not any other operating system as there is one dependency (fsevents) that is needed for macOS and may not be installed if the command is run on a Windows or Linux machine."


Hmmm...tried this on my windows vm and I'm seeing the same behaviour. I think we can punt this for now. It's documented at least. Thanks for digging into this.

edoardopirovano requested a review from aeisenberg June 1, 2021 13:25

edoardopirovano force-pushed the db-upload-deps branch 3 times, most recently from 1bd793f to 91027e4 Compare June 1, 2021 13:57

aeisenberg reviewed Jun 1, 2021

View reviewed changes

edoardopirovano force-pushed the db-upload-deps branch 2 times, most recently from 943e7b3 to c33f830 Compare June 1, 2021 16:47

aeisenberg reviewed Jun 1, 2021

View reviewed changes

src/toolrunner-error-catcher.test.ts Outdated Show resolved Hide resolved

edoardopirovano force-pushed the db-upload-deps branch from c33f830 to 5b7c3a4 Compare June 1, 2021 17:04

aeisenberg reviewed Jun 1, 2021

View reviewed changes

aeisenberg approved these changes Jun 1, 2021

View reviewed changes

Add some dependencies for uploading artifacts

9094b62

edoardopirovano force-pushed the db-upload-deps branch from 5b7c3a4 to 9094b62 Compare June 2, 2021 09:02

edoardopirovano enabled auto-merge (rebase) June 2, 2021 09:26

edoardopirovano merged commit 0cbd4b5 into github:main Jun 2, 2021

github-actions bot mentioned this pull request Jun 7, 2021

Merge main into v1 #549

Merged

5 tasks

This was referenced Apr 24, 2023

[Snyk] Fix for 19 vulnerabilities aliscco/codeql-action#61

Open

[Snyk] Fix for 5 vulnerabilities aliscco/codeql-action#62

Open

snyk-bot mentioned this pull request Apr 24, 2023

[Snyk] Fix for 5 vulnerabilities aliscco/codeql-action#64

Open

This was referenced Apr 24, 2023

[Snyk] Security upgrade xo from 0.24.0 to 0.42.0 aliscco/codeql-action#69

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#75

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#76

Open

This was referenced Apr 24, 2023

[Snyk] Fix for 3 vulnerabilities aliscco/codeql-action#83

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#84

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#86

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#87

Open

This was referenced Apr 24, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#88

Open

[Snyk] Security upgrade xo from 0.24.0 to 0.42.0 aliscco/codeql-action#89

Open

snyk-bot mentioned this pull request Apr 24, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#90

Open

aliscco mentioned this pull request Apr 28, 2023

[Snyk] Fix for 3 vulnerabilities aliscco/codeql-action#386

Open

snyk-bot mentioned this pull request Apr 29, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#390

Open

This was referenced Apr 29, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#393

Open

[Snyk] Fix for 8 vulnerabilities aliscco/codeql-action#397

Open

snyk-bot mentioned this pull request Apr 29, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#400

Open

This was referenced Apr 29, 2023

[Snyk] Fix for 17 vulnerabilities aliscco/codeql-action#403

Open

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#405

Open

snyk-bot mentioned this pull request Apr 29, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#413

Open

aliscco mentioned this pull request Apr 29, 2023

[Snyk] Fix for 4 vulnerabilities aliscco/codeql-action#414

Open

chrisjcox79 mentioned this pull request Jun 20, 2023

[Snyk] Security upgrade semver from 7.3.2 to 7.5.2 chrisjcox79/codeql-action#7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add some dependencies for uploading artifacts #538

Add some dependencies for uploading artifacts #538

edoardopirovano commented Jun 1, 2021 •

edited

Loading

aeisenberg left a comment

aeisenberg Jun 1, 2021

edoardopirovano Jun 1, 2021

aeisenberg Jun 1, 2021

edoardopirovano Jun 1, 2021

edoardopirovano commented Jun 1, 2021 •

edited

Loading

aeisenberg Jun 1, 2021

edoardopirovano Jun 1, 2021 •

edited

Loading

aeisenberg Jun 1, 2021

aeisenberg Jun 1, 2021

	>&2 echo "Failed: node_modules are not up to date. Run 'npm ci' and 'npm run removeNPMAbsolutePaths' on a Mac OS X machine to update"
	>&2 echo "Failed: node_modules are not up to date. Run 'npm ci && npm run removeNPMAbsolutePaths' on a Mac OS X machine to update"

Add some dependencies for uploading artifacts #538

Add some dependencies for uploading artifacts #538

Conversation

edoardopirovano commented Jun 1, 2021 • edited Loading

Merge / deployment checklist

aeisenberg left a comment

Choose a reason for hiding this comment

aeisenberg Jun 1, 2021

Choose a reason for hiding this comment

edoardopirovano Jun 1, 2021

Choose a reason for hiding this comment

aeisenberg Jun 1, 2021

Choose a reason for hiding this comment

edoardopirovano Jun 1, 2021

Choose a reason for hiding this comment

edoardopirovano commented Jun 1, 2021 • edited Loading

aeisenberg Jun 1, 2021

Choose a reason for hiding this comment

edoardopirovano Jun 1, 2021 • edited Loading

Choose a reason for hiding this comment

aeisenberg Jun 1, 2021

Choose a reason for hiding this comment

aeisenberg Jun 1, 2021

Choose a reason for hiding this comment

edoardopirovano commented Jun 1, 2021 •

edited

Loading

edoardopirovano commented Jun 1, 2021 •

edited

Loading

edoardopirovano Jun 1, 2021 •

edited

Loading