Enable Security's Cypress tests on all PRs (#167516)

## Summary

Security Solution writes e2e and other tests using Cypress. In the past,
these tests, if they failed on a tracked branch, couldn't be easily
skipped. They also weren't run in parallelized jobs. For primarily these
reasons, they didn't run on most Kibana PRs.

This PR moves these Cypress tests back to the main PR pipeline. Tests
that fail on tracked branches create (or update) Github issues which can
be used with the skip-test github workflow script to easily skip the
failing tests. The pipeline steps are parallelized and run in under 40
minutes.

### Open Questions

- [ ] Should this PR enable Serverless Security Defend Workflows Cypress
Tests @patrykkopycinski

### Some buildkite pipelines that used to run only on Security PRs now
run on all PRs:

These steps run on all PRs with these changes

- Security Solution Cypress Tests (general tests that haven't been
organized into an area team)
- Explore tests
- Investigations Tests
- Defend Workflows Tests
- Defend Workflows Serverless
- Threat Intel Tests
- OS Query Tests
- Security Solution Burning Changed Specs (these run only recently
changed specs a few extra times)
- Security Solution OpenAPI codegen
- OSQuery burning
- OSQuery Serverless

<details>
  <summary><b>And these already run on all PRs</b></summary>

  - Serverless Security Cypress Tests
  - Serverless Explore tests
  - Serverless Investigations Tests
</details>

### Security Cypress tests run in the main `on merge` pipeline instead
of the `on merge unsupported ftrs` pipeline:

These steps run in the `on merge` pipeline with these changes:

- Security Solution Cypress Tests
- Explore Cypress Tests
- Investigations Cypress Tests
- Defend Workflows Cypress Tests
- Defend Workflows Serverless Cypress Tests
- Threat Intelligence Cypress Tests
- Osquery Cypress Tests

<details>
<summary><b>and these already run on the `on merge`
pipeline</b></summary>

- Serverless Security Cypress Tests
- Serverless Explore - Security Solution Cypress Tests
- Serverless Investigations - Security Solution Cypress Tests

</details>

### Additional work to be done:

We need to consolidate build steps, enhance test skipping to support
Cypress-grep flags, avoid out-of-memory errors in cypress, enhance
parallelization, improve Cypress reporting, and probably other things.
These are tracked separately. Reach out to me if you need details.

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

.buildkite/pipelines/on_merge.yml

@@ -115,6 +115,90 @@ steps:
         - exit_status: '*'
           limit: 1
 
+  - command: .buildkite/scripts/steps/functional/security_solution.sh
+    label: 'Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 16
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/security_solution_explore.sh
+    label: 'Explore - Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 4
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
+    label: 'Investigations - Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 8
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/defend_workflows.sh
+    label: 'Defend Workflows Cypress Tests'
+    agents:
+      queue: n2-4-virt
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 6
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
+    label: 'Defend Workflows Cypress Tests on Serverless'
+    agents:
+      queue: n2-4-virt
+    depends_on: build
+    timeout_in_minutes: 120
+    parallelism: 2
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/threat_intelligence.sh
+    label: 'Threat Intelligence Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 120
+    parallelism: 2
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/osquery_cypress.sh
+    label: 'Osquery Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 50
+    parallelism: 6
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
   - command: '.buildkite/scripts/steps/functional/on_merge_unsupported_ftrs.sh'
     label: Trigger unsupported ftr tests
     timeout_in_minutes: 10

.buildkite/pipelines/on_merge_unsupported_ftrs.yml

@@ -49,66 +49,6 @@ steps:
         - exit_status: '*'
           limit: 1
 
-  - command: .buildkite/scripts/steps/functional/security_solution.sh
-    label: 'Security Solution Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 120
-    parallelism: 14
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-
-  - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
-    label: 'Investigations - Security Solution Cypress Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 120
-    parallelism: 6
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-
-  - command: .buildkite/scripts/steps/functional/security_solution_explore.sh
-    label: 'Explore - Security Solution Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 60
-    parallelism: 2
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-
-  - command: .buildkite/scripts/steps/functional/defend_workflows.sh
-    label: 'Defend Workflows Cypress Tests'
-    agents:
-      queue: n2-4-virt
-    depends_on: build
-    timeout_in_minutes: 120
-    parallelism: 6
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-
-  - command: .buildkite/scripts/steps/functional/osquery_cypress.sh
-    label: 'Osquery Cypress Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 50
-    parallelism: 6
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-
   - command: .buildkite/scripts/steps/functional/synthetics_plugin.sh
     label: 'Synthetics @elastic/synthetics Tests'
     agents:

.buildkite/pipelines/pull_request/base.yml

@@ -93,6 +93,136 @@ steps:
         - exit_status: '*'
           limit: 1
 
+  - command: .buildkite/scripts/steps/functional/security_solution.sh
+    label: 'Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 16
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/security_solution_explore.sh
+    label: 'Explore - Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 4
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
+    label: 'Investigations - Security Solution Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 8
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/defend_workflows.sh
+    label: 'Defend Workflows Cypress Tests'
+    agents:
+      queue: n2-4-virt
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 6
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
+    label: 'Defend Workflows Cypress Tests on Serverless'
+    agents:
+      queue: n2-4-virt
+    depends_on: build
+    timeout_in_minutes: 120
+    parallelism: 2
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/threat_intelligence.sh
+    label: 'Threat Intelligence Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 120
+    parallelism: 2
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/osquery_cypress.sh
+    label: 'Osquery Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 50
+    parallelism: 6
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+    artifact_paths:
+      - 'target/kibana-osquery/**/*'
+
+  - command: .buildkite/scripts/steps/functional/security_solution_burn.sh
+    label: 'Security Solution Cypress tests, burning changed specs'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 60
+    parallelism: 1
+    retry:
+      automatic: false
+    soft_fail: true
+
+  - command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
+    label: 'Security Solution OpenAPI codegen'
+    agents:
+      queue: n2-2-spot
+    timeout_in_minutes: 60
+    parallelism: 1
+
+  - command: .buildkite/scripts/steps/functional/osquery_cypress_burn.sh
+    label: 'Osquery Cypress Tests, burning changed specs'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 50
+    soft_fail: true
+    retry:
+      automatic: false
+    artifact_paths:
+      - 'target/kibana-osquery/**/*'
+
+  - command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh
+    label: 'Serverless Osquery Cypress Tests'
+    agents:
+      queue: n2-4-spot
+    depends_on: build
+    timeout_in_minutes: 50
+    parallelism: 6
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
+    artifact_paths:
+      - 'target/kibana-osquery/**/*'
+
   # status_exception: Native role management is not enabled in this Elasticsearch instance
   # - command: .buildkite/scripts/steps/functional/security_serverless_defend_workflows.sh
   #   label: 'Serverless Security Defend Workflows Cypress Tests'