Cloud runner develop - Stabilizes kubernetes provider (game-ci#531)

* fixes * fixes * fixes * fixes * fixes * check for startup message in workflows * check for startup message in workflows * check for startup message in workflows * check for startup message in workflows * check for startup message in workflows * check for startup message in workflows * Update cloud-runner-ci-pipeline.yml * Update cloud-runner-ci-pipeline.yml * no storage class specified * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * updates * log file path * latest develop * log file path * log file path * Update package.json * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * log file path * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * stream logs through standard input and new remote client cli command * update pipeline to use k3s * version: 'latest' * fixes * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * disable aws pipe for now * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * push k8s logs to LOG SERVICE IP * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * tests * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * podname logs for log service * hashed logs * hashed logs * hashed logs * hashed logs * hashed logs * hashed logs * no wait, just repeat logs * no wait, just repeat logs * remove typo - double await * test fix - kubernetes - name typo in github yaml * test fix - kubernetes - name typo in github yaml * check missing log file * check missing log file * Push to steam test * Push to steam test * Fix path * k8s reliable log hashing * k8s reliable log hashing * k8s reliable log hashing * hashed logging k8s * hashed logging k8s * hashed logging k8s * hashed logging k8s * hashed logging k8s * hashed logging k8s * Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line * Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line * Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line * Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line * Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line * Fix exit flow for k8s job * hash comparison logging for log complete in k8s flow * Interrupt k8s logs when logs found * cleanup async parameter * cleanup async parameter * cleanup async parameter * fixes * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix
homagames · Mar 7, 2024 · 184e3ce · 184e3ce
1 parent 8708814
commit 184e3ce
Show file tree

Hide file tree

Showing 47 changed files with 1,455 additions and 855 deletions.
diff --git a/.eslintrc.json b/.eslintrc.json
@@ -1,22 +1,11 @@
 {
-  "plugins": [
-    "jest",
-    "@typescript-eslint",
-    "prettier",
-    "unicorn"
-  ],
-  "extends": [
-    "plugin:unicorn/recommended",
-    "plugin:github/recommended",
-    "plugin:prettier/recommended"
-  ],
+  "plugins": ["jest", "@typescript-eslint", "prettier", "unicorn"],
+  "extends": ["plugin:unicorn/recommended", "plugin:github/recommended", "plugin:prettier/recommended"],
   "parser": "@typescript-eslint/parser",
   "parserOptions": {
     "ecmaVersion": 2020,
     "sourceType": "module",
-    "extraFileExtensions": [
-      ".mjs"
-    ],
+    "extraFileExtensions": [".mjs"],
     "ecmaFeatures": {
       "impliedStrict": true
     },
@@ -33,10 +22,7 @@
     // Namespaces or sometimes needed
     "import/no-namespace": "off",
     // Properly format comments
-    "spaced-comment": [
-      "error",
-      "always"
-    ],
+    "spaced-comment": ["error", "always"],
     "lines-around-comment": [
       "error",
       {
@@ -71,12 +57,7 @@
     // Enforce camelCase
     "camelcase": "error",
     // Allow forOfStatements
-    "no-restricted-syntax": [
-      "error",
-      "ForInStatement",
-      "LabeledStatement",
-      "WithStatement"
-    ],
+    "no-restricted-syntax": ["error", "ForInStatement", "LabeledStatement", "WithStatement"],
     // Continue is viable in forOf loops in generators
     "no-continue": "off",
     // From experience, named exports are almost always desired. I got tired of this rule

diff --git a/.github/workflows/cloud-runner-ci-pipeline.yml b/.github/workflows/cloud-runner-ci-pipeline.yml
@@ -18,42 +18,37 @@ env:
   GCP_PROJECT: unitykubernetesbuilder
   GCP_LOG_FILE: ${{ github.workspace }}/cloud-runner-logs.txt
   AWS_REGION: eu-west-2
-  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
-  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
   AWS_DEFAULT_REGION: eu-west-2
   AWS_STACK_NAME: game-ci-team-pipelines
   CLOUD_RUNNER_BRANCH: ${{ github.ref }}
   DEBUG: true
-  UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
+  UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
+  UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
+  UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
   PROJECT_PATH: test-project
   UNITY_VERSION: 2019.3.15f1
   USE_IL2CPP: false
   USE_GKE_GCLOUD_AUTH_PLUGIN: true
-  GIT_PRIVATE_TOKEN: ${{ secrets.GIT_PRIVATE_TOKEN }}
-  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
 jobs:
-  smokeTests:
-    name: Smoke Tests
+  tests:
+    name: Tests
     if: github.event.event_type != 'pull_request_target'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         test:
-          #- 'cloud-runner-async-workflow'
+          - 'cloud-runner-end2end-locking'
+          - 'cloud-runner-end2end-caching'
+          - 'cloud-runner-end2end-retaining'
           - 'cloud-runner-caching'
-          # - 'cloud-runner-end2end-caching'
-          # - 'cloud-runner-end2end-retaining'
           - 'cloud-runner-environment'
+          - 'cloud-runner-image'
           - 'cloud-runner-hooks'
           - 'cloud-runner-local-persistence'
           - 'cloud-runner-locking-core'
           - 'cloud-runner-locking-get-locked'
-        providerStrategy:
-          #- aws
-          - local-docker
-          #- k8s
     steps:
       - name: Checkout (default)
         uses: actions/checkout@v4
@@ -65,58 +60,85 @@ jobs:
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: eu-west-2
-      - uses: google-github-actions/auth@v1
-        if: matrix.providerStrategy == 'k8s'
+      - run: yarn
+      - run: yarn run test "${{ matrix.test }}" --detectOpenHandles --forceExit --runInBand
+        timeout-minutes: 60
+        env:
+          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
+          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
+          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
+          PROJECT_PATH: test-project
+          TARGET_PLATFORM: StandaloneWindows64
+          cloudRunnerTests: true
+          versioning: None
+          KUBE_STORAGE_CLASS: local-path
+          PROVIDER_STRATEGY: local-docker
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          GIT_PRIVATE_TOKEN: ${{ secrets.GIT_PRIVATE_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - run: |
+          docker volume rm $(docker volume ls -qf dangling=true)
+          docker rmi $(docker images -q -f dangling=true)
+          docker rm $(docker ps -aqf status=exited)
+          docker rmi $(docker images -q)
+  k8sTests:
+    name: K8s Tests
+    if: github.event.event_type != 'pull_request_target'
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        test:
+          # - 'cloud-runner-async-workflow'
+          - 'cloud-runner-end2end-locking'
+          - 'cloud-runner-end2end-caching'
+          - 'cloud-runner-end2end-retaining'
+          - 'cloud-runner-kubernetes'
+          - 'cloud-runner-environment'
+          - 'cloud-runner-github-checks'
+    steps:
+      - name: Checkout (default)
+        uses: actions/checkout@v2
         with:
-          credentials_json: ${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}
-      - name: 'Set up Cloud SDK'
-        if: matrix.providerStrategy == 'k8s'
-        uses: 'google-github-actions/setup-gcloud@v1.1.0'
-      - name: Get GKE cluster credentials
-        if: matrix.providerStrategy == 'k8s'
-        run: |
-          export USE_GKE_GCLOUD_AUTH_PLUGIN=True
-          gcloud components install gke-gcloud-auth-plugin
-          gcloud container clusters get-credentials $GKE_CLUSTER --zone $GKE_ZONE --project $GKE_PROJECT
+          lfs: false
       - run: yarn
+      - name: actions-k3s
+        uses: debianmaster/actions-k3s@v1.0.5
+        with:
+          version: 'latest'
       - run: yarn run test "${{ matrix.test }}" --detectOpenHandles --forceExit --runInBand
-        timeout-minutes: 35
+        timeout-minutes: 60
         env:
-          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
+          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
+          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
+          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
           PROJECT_PATH: test-project
           TARGET_PLATFORM: StandaloneWindows64
           cloudRunnerTests: true
           versioning: None
-          CLOUD_RUNNER_CLUSTER: ${{ matrix.providerStrategy }}
-  tests:
-    # needs:
-    #   - smokeTests
-    #   - buildTargetTests
-    name: Integration Tests
+          KUBE_STORAGE_CLASS: local-path
+          PROVIDER_STRATEGY: k8s
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          GIT_PRIVATE_TOKEN: ${{ secrets.GIT_PRIVATE_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  awsTests:
+    name: AWS Tests
     if: github.event.event_type != 'pull_request_target'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        providerStrategy:
-          - aws
-          - local-docker
-          - k8s
         test:
-          - 'cloud-runner-async-workflow'
-          #- 'cloud-runner-caching'
           - 'cloud-runner-end2end-locking'
           - 'cloud-runner-end2end-caching'
           - 'cloud-runner-end2end-retaining'
           - 'cloud-runner-environment'
-          #- 'cloud-runner-hooks'
           - 'cloud-runner-s3-steps'
-          #- 'cloud-runner-local-persistence'
-          #- 'cloud-runner-locking-core'
-          #- 'cloud-runner-locking-get-locked'
     steps:
       - name: Checkout (default)
-        uses: actions/checkout@v4
+        uses: actions/checkout@v2
         with:
           lfs: false
       - name: Configure AWS Credentials
@@ -125,29 +147,24 @@ jobs:
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: eu-west-2
-      - uses: google-github-actions/auth@v1
-        if: matrix.providerStrategy == 'k8s'
-        with:
-          credentials_json: ${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}
-      - name: 'Set up Cloud SDK'
-        if: matrix.providerStrategy == 'k8s'
-        uses: 'google-github-actions/setup-gcloud@v1.1.0'
-      - name: Get GKE cluster credentials
-        if: matrix.providerStrategy == 'k8s'
-        run: |
-          export USE_GKE_GCLOUD_AUTH_PLUGIN=True
-          gcloud components install gke-gcloud-auth-plugin
-          gcloud container clusters get-credentials $GKE_CLUSTER --zone $GKE_ZONE --project $GKE_PROJECT
       - run: yarn
       - run: yarn run test "${{ matrix.test }}" --detectOpenHandles --forceExit --runInBand
         timeout-minutes: 60
         env:
-          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
+          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
+          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
+          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
           PROJECT_PATH: test-project
           TARGET_PLATFORM: StandaloneWindows64
           cloudRunnerTests: true
           versioning: None
-          PROVIDER_STRATEGY: ${{ matrix.providerStrategy }}
+          KUBE_STORAGE_CLASS: local-path
+          PROVIDER_STRATEGY: aws
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          GIT_PRIVATE_TOKEN: ${{ secrets.GIT_PRIVATE_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
   buildTargetTests:
     name: Local Build Target Tests
     runs-on: ubuntu-latest
@@ -164,7 +181,7 @@ jobs:
           - StandaloneLinux64 # Build a Linux 64-bit standalone.
           - WebGL # WebGL.
           - iOS # Build an iOS player.
-          - Android # Build an Android .apk.
+          # - Android # Build an Android .apk.
     steps:
       - name: Checkout (default)
         uses: actions/checkout@v4
@@ -175,7 +192,13 @@ jobs:
         id: unity-build
         timeout-minutes: 30
         env:
-          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
+          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
+          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
+          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          GIT_PRIVATE_TOKEN: ${{ secrets.GIT_PRIVATE_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
           cloudRunnerTests: true
           versioning: None
@@ -188,3 +211,8 @@ jobs:
           name: ${{ matrix.providerStrategy }} Build (${{ matrix.targetPlatform }})
           path: ${{ steps.unity-build.outputs.BUILD_ARTIFACT }}
           retention-days: 14
+      - run: |
+          docker volume rm $(docker volume ls -qf dangling=true)
+          docker rmi $(docker images -q -f dangling=true)
+          docker rm $(docker ps -aqf status=exited)
+          docker rmi $(docker images -q)
diff --git a/action.yml b/action.yml
@@ -186,11 +186,11 @@ inputs:
     description:
       '[CloudRunner] Either local, k8s or aws can be used to run builds on a remote cluster. Additional parameters must
       be configured.'
-  cloudRunnerCpu:
+  containerCpu:
     default: ''
     required: false
     description: '[CloudRunner] Amount of CPU time to assign the remote build container'
-  cloudRunnerMemory:
+  containerMemory:
     default: ''
     required: false
     description: '[CloudRunner] Amount of memory to assign the remote build container'