Speed up CI by migrating to Yarn 3 (#248)

* improve lunasec plug * improve iphone words * clean up the secondary attack vector explanation * a word * update docusaurus to fix blogs and fix the awful looking admonition from beta 12 * clearer warning about dnslog.cn * remove sentence saying there was no stable release, no longer relevant * Better CTAs on the page * Add social info * Remove random stuff * Fix example to use log4j2 pedantry: make the example code actually valid log4j2 use, not log4j 1.x * make demo backend use pm2 * Update 2021-12-09-log4j-zero-day.md * Slightly improve example code * log4j download page (#269) Apache projects' primary approach to releases is to provide downloads via the apache project web sites - not via github. * update yarn.lock * Regenerate lockfile, fix CLI arguments for hosted demo * Disable Nginx volume * Fix a yarn error * Fix yarn.lock * Remove extraneous workdir statement * Uncomment nginx * Remove second build step from demo back end build * Revert build change * no fork pm2 * switch to pm2-runtime * change nolookups compatibility The `nolookups` work-around mentioned only works on versions >= 2.7. Fixes #274. * Update log4j post title * Add log4shell CLI tool * Fix bad path * Fix entrypoint for package * Change version to beta * Fix script to work with both a specific path or in the current folder * WIP blog post * Bump version * make hash downloading automatic even if not using NPM * also find war files * move log4shell to tools * improve DNS test paragraph * Update 2021-12-09-log4j-zero-day.md * more small post edits like date, forrest as an author * get date out of title because title too long and date updated properly now * log4shell scanning cli initial commit * Enabled options for printing out json for parsing results. * Add option to write outputs to a file. * update binary name to log4shell * Write up the rest of the blog post * when scanning archives, scan nested ones * Wrap up the Log4Shell Mitigation Guide doc * More post cleanup * More post cleanup * Fix Master CI * Fix grammar in mitigation guide * Fix bad link in blog post * Fix typo * blog edits to header example * Add social links and update main Readme * Remove thank you line * big mitigation edits * Add CVE number back to first line of text for SEO * Content reworking * Add log4j to first sentence * few tiny edits * small edits linking two blog posts together and other nits * Mitigation edits forrest (#295) * big mitigation edits * Add CVE number back to first line of text for SEO * Content reworking * Add log4j to first sentence * few tiny edits * small edits linking two blog posts together and other nits Co-authored-by: Free Wortley <free@lunasec.io> * Adding command for running log4shell hotpatch server. The command brings up the servers, but they currently do not work. * fix package mistake * mention log4j 2.16 * add contact form, what a doozy * remove bad dep and eslint ignore something * add mui types * verbose start in CI * yarn install * made bucket script wait for file and brought back the use of a precache container * switch cli workdir to repo * Hotpatching works when being tested locally again vulnerable spring server. * Fix renamed directory * add warnings about 2.15 and flag * better warning * fix typo and add CVE name * more CVE mentions * Update 2021-12-09-log4j-zero-day.md * Update 2021-12-09-log4j-zero-day.md * fix english (#304) * Add new blog post on 2nd log4j vulnerability * update Log4ShellHotpatch * Add updated dates * Add disclaimer about log format still being vulnerable * Add disclaimer about log format still being vulnerable * Added post content for follow up CVE under certain circumstances * Update times in doc * Scanner finds 2.15 (#305) * first draft of adding severity rating to vulns * duplicate flags onto scan command because its more natural UX * added 2.15 hashes and confirmed they work * Update vulnerablehashes.go * Update vulnerablehashes.go * Severity 9.8 for log4j v1 vulns * Swap from Severity to CVE * put severity back in Co-authored-by: Johnathan Free Wortley <free@lunasec.io> * Cleanup content * update some wording in the blog post * blog mentions hot patch cli * Update 2021-12-12-log4j-zero-day-mitigation-guide.mdx * Change links to the generic Releases page * print payload string from CLI * prettier output * bump version * make goreleaser just do binaries * added more options to the hotpatch server and added a landing page * update hotpatch server to have more descriptive text * Update Patch section with new notes * Update timestamps * Wordsmithing * add live patch blog post * Fix image links to be persistent * Fix image link for bad image also * Fix bad image links by using MDX syntax instead * change dependency to not panic * bump version of log4shell cli * add docker-compose and update readme with some commands * update blog posts * Blog post updates * Fix formatting * Tweaks * fix typo * scan library before browsing it * Fix some typos * Better phrasing * feat: scan into zip archives in addition to jar+war * script for downloading all log4j versions * try again in CI * add payload url to the print out in the cli * update blog post to fix changes suggested in issues * CLI UX improvements and more legalish warnings * use webarchive to reference zero day tweet * increase max mocked s3 body size in nginx for live demo * fix nginx args * Basic technical analysis of the Log4Shell exploit * pull all maven and apache versions of log4j * update blog to include java decomp * log4shell and 2.15.0 cves are distinct in findings now * bump version * add zip and ear extensions to allow deep scans * include 1.2.17 in scanning log4j1 * bump version * bump cli version to 1.3.2 * warning about virus scanners in blog post * resolve symlinks while scanning * switch all logs to stdout and prettier formatting for scan results * Add links back to other posts * slightly better log level printing * Add links to other blog posts and update phrasing * update CTA size * Add FUNDING.yml file for GitHub Sponsors * Update README.md * add manual releasing instructions * fix false positive for 2.16.0 and 2.15.0 * analyzer has better semver version checking * improve log colors * version change is more than a patch, version should reflect this * global flags are recognized by the cli if they have a name collision in a subcommand * create blog post discussing follow up issues for cve * add --no-follow-symlinks * increase severity of cve-2021-45046 finding * add details about the latest updates about the log4shell cves * update date * Fix bug in the new CVSS post * One more change * Add bypass payload to post * Fix bad date * Update issue templates * broken symlinks no longer stop scanning * bump version * WIP OSS patching blog post (#348) * WIP OSS patching blog post * small post edits * oss patching blog post drafted * update date and truncate * get license out of PR template and add to ignore file * explain more why people need to know to security patch * update intro wording * edits to include githubs tooling * more credit to google * nits * change a link Co-authored-by: breadchris <chris@lunasec.io> * typo 'and' should be 'an' * update guidance to use 2.17.0 * Update guidance across all posts * better osx instructions * Update 2021-12-12-log4j-zero-day-mitigation-guide.mdx * Update 2021-12-12-log4j-zero-day-mitigation-guide.mdx * Update the malicious links to be our domain everywhere * close read which is left open * fix admonitions * yarn stuff * include more install steps in the precache * stop demo back end from starting prematurely * delete &&, it wasnt needed * rename integration test workflow to just test Co-authored-by: Forrest <forrest@lunasec.io> Co-authored-by: Stu Tomlinson <stu@nosnilmot.com> Co-authored-by: Moya <moya@asofterspace.com> Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com> Co-authored-by: Thompson, Brian <foss.systems@icloud.com> Co-authored-by: breadchris <chris@lunasec.io> Co-authored-by: Sebastian Lövdahl <slovdahl@hibox.fi> Co-authored-by: Pascal Verdage <pascal.verdage@sgcib.com> Co-authored-by: Dan Hoizner <dan.hoizner@collibra.com> Co-authored-by: Alex Collignon <colligno@adobe.com> Co-authored-by: Forrest <light24bulbs@gmail.com> Co-authored-by: Tobi Lehman <mail@tobilehman.com> Former-commit-id: a503b02 Former-commit-id: ee712a38177db4be394be0634d3c82fccb223a2e
lunasec-io · Dec 21, 2021 · 627c520 · 627c520
1 parent 4e2e80f
commit 627c520
Show file tree

Hide file tree

Showing 227 changed files with 49,501 additions and 30,414 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -2,8 +2,6 @@
 .idea/
 .github/
 
-docs/
-
 docker-compose.yaml
 docker-compose.*.yaml
 
@@ -12,4 +10,4 @@ outputs/
 **/build/
 **/node_modules/
 node_modules/
-.npmrc
+.npmrc
diff --git a/.editorconfig b/.editorconfig
@@ -0,0 +1,10 @@
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+
+[*.{js,json,yml}]
+charset = utf-8
+indent_style = space
+indent_size = 2
diff --git a/.eslintignore b/.eslintignore
@@ -4,3 +4,9 @@
 **/build
 **/generated
 js/demo-apps/packages/react-front-end/cypress/integration/secure_components_spec.ts
+.pnp.cjs
+*.mjs
+deploy-apigateway-to-firehose.ts
+**/*.vue
+**/metrics-server-backend/*
+**/ContactForm.jsx
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -21,10 +21,10 @@ module.exports = {
     node: true
   },
   extends: [
-    "plugin:vue/vue3-essential",
-    "@vue/typescript/recommended",
-    "@vue/prettier",
-    "@vue/prettier/@typescript-eslint",
+    // "plugin:vue/vue3-essential",
+    // "@vue/typescript/recommended",
+    // "@vue/prettier",
+    // "@vue/prettier/@typescript-eslint",
     'eslint:recommended',
     'plugin:@typescript-eslint/recommended',
     'plugin:@typescript-eslint/recommended-requiring-type-checking',
@@ -54,10 +54,12 @@ module.exports = {
   },
   plugins: [
     'react',
-    '@typescript-eslint'
+    '@typescript-eslint',
+    'jest'
   ],
   rules: {
-    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off',
+    "@typescript-eslint/no-unsafe-argument": 1, // TODO: Re-enable this rule and fix all errors
+    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off', // These never error, currently
     'no-debugger': process.env.NODE_ENV === 'production' ? 'error' : 'off',
     eqeqeq: 'error',
     quotes: ['error', 'single', { allowTemplateLiterals: true, avoidEscape: true }],

diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml
@@ -0,0 +1,2 @@
+github: lunasec-io
+custom: "https://www.lunasec.io/contact"
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,40 @@
+<!--
+  ~ Copyright by LunaSec (owned by Refinery Labs, Inc)
+  ~
+  ~ Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
+  ~ (the "License"); you may not use this file except in compliance with the
+  ~ License. You may obtain a copy of the License at
+  ~
+  ~ https://creativecommons.org/licenses/by-sa/4.0/legalcode
+  ~
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+-->
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**STOP**: Is this a **security vulnerability**?  If so, follow Responsible Disclosure and email us at security@lunasec.io instead of opening an issue.  
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.
diff --git a/.github/workflows/check-deps.yaml b/.github/workflows/check-deps.yaml
@@ -0,0 +1,23 @@
+name: Check Dependencies
+
+# This workflow checks to make sure that all dependency binaries are valid, to prevent a sneaky commit doing something malicious
+on:
+  pull_request:
+    branches: ['**', '**']
+  push:
+    branches: [master]
+
+jobs:
+  check-deps:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    - name: dependencies
+      env:
+        IS_LUNASEC_CI: true
+      run: yarn install --immutable --immutable-cache --check-cache --inline-builds
diff --git a/.github/workflows/composite/merge-branch/action.yml b/.github/workflows/composite/merge-branch/action.yml
@@ -0,0 +1,22 @@
+name: 'Merge Master Branch'
+description: 'Merges the master Git branch into the current branch.'
+
+runs:
+  using: "composite"
+  steps:
+    # TODO: there may be a more idiomatic way to do this
+    - name: Check if we should skip this build
+      shell: bash
+      id: job-canceller
+      run: echo "::set-output name=cancelled::${{ github.event_name != 'pull_request' }}"
+
+     # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
+    - name: cancelling
+      uses: andymckay/cancel-action@0.2
+      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
+
+    # Merge with master (or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
+    - name: Merge target branch
+      shell: bash
+      run: git merge origin/${{ github.event.pull_request.base.ref }}
+      if: ${{ github.event_name == 'pull_request' }}
diff --git a/.github/workflows/composite/setup-docker-ci/action.yml b/.github/workflows/composite/setup-docker-ci/action.yml
@@ -0,0 +1,40 @@
+name: 'Setup Docker CI Environment'
+description: 'Creates the LunaSec CI build environment'
+
+inputs:
+  merge_master:
+    description: 'If true, this merges the master branch during setup.'
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Docker BuildKit
+      id: buildx
+      uses: docker/setup-buildx-action@v1
+      with:
+        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
+        driver: docker
+
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    # specifically tag this bootstrap container to prevent rebuilds
+    - name: Build Lerna Bootstrap container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t lerna-bootstrap --target lerna-bootstrap .
+
+    - name: Build CLI Container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .
diff --git a/.github/workflows/documentation.yaml b/.github/workflows/documentation.yaml
@@ -41,19 +41,13 @@ jobs:
         with:
           node-version: '16'
 
-      # Speed up builds by caching node_modules
-      - uses: actions/cache@v2
-        with:
-          path: |
-            node_modules
-            */*/node_modules
-          key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
-
       - name: Lerna Bootstrap and Build Deploy Tool
+        env:
+          IS_LUNASEC_CI: true
+          CI: true
         run: |
-          npx lerna bootstrap --ci
+          yarn install --immutable --immutable-cache --inline-builds
           yarn compile:dev:infrastructure
-          npx lerna link
 
       - name: Build Repo
         working-directory: js/sdks

diff --git a/.github/workflows/integration.yaml b/.github/workflows/integration.yaml
@@ -14,7 +14,7 @@
 #
 # This is a basic workflow to help you get started with Actions
 
-name: CI
+name: Tests
 
 # Controls when the workflow will run
 on:
@@ -36,53 +36,27 @@ jobs:
     env:
       # avoid warnings like "tput: No value for $TERM and no -T specified"
       TERM: xterm
-      RUNNING_IN_CI: true
       DOCKER_BUILDKIT: 1
       COMPOSE_DOCKER_CLI_BUILD: 1
     strategy:
       matrix:
         merge: ["merged", "not-merged"]
-        lockfile: ["lockfile","no-lockfile"]
+        lockfile: ["lockfile"] #,"no-lockfile"]
         exclude:
           - merge: "merged"
             lockfile: "no-lockfile"
 
     steps:
-
     - uses: actions/checkout@v2
       with:
         fetch-depth: 0
 
-    #TODO: there may be a more idiomatic way to do this
-    - name: Check if we should skip this build
-      id: job-canceller
-      run: echo "::set-output name=cancelled::${{ matrix.merge == 'merged' && github.event_name != 'pull_request'}}"
-
-    # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
-    - name: cancelling
-      uses: andymckay/cancel-action@0.2
-      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
-
-      # merge with master(or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
-    - name: Merge target branch
-      run: git merge origin/${{ github.event.pull_request.base.ref }}
-      if: ${{ matrix.merge == 'merged' && github.event_name == 'pull_request' }}
-
-    - uses: actions/setup-node@v2
-      with:
-        node-version: '14'
-        cache: yarn
-
-    - uses: actions/setup-go@v2
-      with:
-        go-version: '^1.17.1'
+    - name: Setup Branch
+      uses: ./.github/workflows/composite/merge-branch
+      if: ${{ matrix.merge == 'merged' }}
 
-    - name: Set up Docker BuildKit
-      id: buildx
-      uses: docker/setup-buildx-action@v1
-      with:
-        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
-        driver: docker
+    - name: Setup Environment
+      uses: ./.github/workflows/composite/setup-docker-ci
 
     # Enable tmate debugging of manually-triggered workflows if the input option was provided
     - name: Setup tmate session
@@ -92,9 +66,13 @@ jobs:
       if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled != 'false' }}
       timeout-minutes: 15
 
-    - name: Delete Lockfile
-      if: ${{ matrix.lockfile == 'no-lockfile' }}
-      run: rm yarn.lock
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+#    - name: Delete and Regenerate Lockfile
+#      if: ${{ matrix.lockfile == 'no-lockfile' }}
+#      run: rm yarn.lock && CI="" yarn install #currently broken
 
 #    - uses: actions/setup-go@v2
 #      with:
@@ -116,22 +94,8 @@ jobs:
 #        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 #      run: npm run license:check
 
-    # TODO: move this into docker, using the same container that the dedicated tokenizer uses
-    - name: Go Tests
-      run: yarn run test:unit:go
-
-    - name: Build CLI Container
-      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .
-
-    # Hijack the cli image to quickly run linting and unit testing
-    - name: Lint
-      run: docker run --entrypoint yarn repo_lunasec-cli lint
-
-    - name: Unit Test
-      run: docker run --entrypoint yarn repo_lunasec-cli test:unit:js
-
     - name: Use CLI Container with docker.sock mounted to launch all other containers
-      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests
+      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests --verbose
 
     - name: docker logs
       if: ${{ always() && steps.job-canceller.outputs.cancelled == 'false' }}
@@ -146,16 +110,50 @@ jobs:
       if: ${{ job.status == 'failure' && matrix.lockfile == 'no-lockfile' }}
       run: echo "::warning  title=lockfile-check::Building without a lockfile failed"
 
-    - uses: actions/upload-artifact@v2
-      if: ${{ always() && steps.job-canceller.outputs.cancelled == 'false' }}
-      with:
-        name: cypress-recording
-        path: /videos/secure_components_spec.ts.mp4
+#    - uses: actions/upload-artifact@v2
+#      if: ${{ always() && steps.job-canceller.outputs.cancelled == 'false' }}
+#      with:
+#        name: cypress-recording
+#        path: /videos/secure_components_spec.ts.mp4
 
     - name: Slack Notification
       uses: rtCamp/action-slack-notify@v2
       if: ${{ steps.job-canceller.outputs.cancelled == 'false' }}
       env:
         SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
         SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
-        SLACK_COLOR: ${{ job.status }}
+        SLACK_COLOR: ${{ job.status }}
+
+  golang-unit-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        merge: [ "merged", "not-merged" ]
+        lockfile: [ "lockfile" ] #,"no-lockfile"]
+        exclude:
+          - merge: "merged"
+            lockfile: "no-lockfile"
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Setup Branch
+        uses: ./.github/workflows/composite/merge-branch
+        if: ${{ matrix.merge == 'merged' }}
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '^1.17.1'
+
+      # TODO: move this into docker, using the same container that the dedicated tokenizer uses
+      - name: Go Tests
+        run: yarn run test:unit:go
+
+      - name: Slack Notification
+        uses: rtCamp/action-slack-notify@v2
+        if: ${{ steps.job-canceller.outputs.cancelled == 'false' }}
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
+          SLACK_COLOR: ${{ job.status }}
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		github: lunasec-io
		custom: "https://www.lunasec.io/contact"