forked from valkey-io/valkey-glide
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1 from Yury-Fridlyand/dbg/yuryf-ci-fork
Dbg/yuryf ci fork
- Loading branch information
Showing
13 changed files
with
424 additions
and
609 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| # CI/CD Workflow Guide | ||
|
|
||
| TODO: Add a description of the CI/CD workflow and its components. | ||
|
|
||
| ### Overview | ||
|
|
||
| Our CI/CD pipeline tests and builds our project across multiple languages, versions, and environments. This guide outlines the key components and processes of our workflow. | ||
|
|
||
| ### Workflow Triggers | ||
|
|
||
| * Push to `main` branch | ||
| * Pull requests | ||
| * Scheduled runs (daily) | ||
| * Manual trigger (workflow_dispatch) | ||
|
|
||
| ### Language-Specific Workflows | ||
|
|
||
| Each language has its own workflow file with similar structure but language-specific steps, for example python.yml for Python, or java.yml for Java. | ||
|
|
||
| ### Shared Components | ||
|
|
||
| #### Matrix Files | ||
|
|
||
| While workflows are language-specific, the matrix files are shared across all workflows. | ||
| Workflows are starting by loading the matrix files from the `.github/json_matrices` directory. | ||
|
|
||
| * `engine-matrix.json`: Defines the versions of the engine to test against. | ||
| * `build-matrix.json`: Defines the host environments for testing. | ||
| * `supported-language-versions.json`: Defines the supported versions of languages. | ||
|
|
||
| All matrices have a `run` like field which specifies if the configuration should be tested on every workflow run. | ||
| This allows for flexible control over which configurations are tested in different scenarios, optimizing CI/CD performance and resource usage. | ||
|
|
||
| #### Engine Matrix (engine-matrix.json) | ||
|
|
||
| Defines the versions of Valkey engine to test against: | ||
|
|
||
| ```json | ||
| [ | ||
| { "type": "valkey", "version": "7.2.5", "run": "always" }, | ||
| // ... other configurations | ||
| ] | ||
| ``` | ||
|
|
||
| * `type`: The type of engine (e.g., Valkey, Redis). | ||
| * `version`: Specifies the engine version that the workflow should checkout. | ||
| For example, "7.2.5" represents a release tag, while "7.0" denotes a branch name. The workflow should use this parameter to checkout the specific release version or branch to build the engine with the appropriate version. | ||
| * `run`: Specifies if the engine version should be tested on every workflow. | ||
|
|
||
| #### Build Matrix (build-matrix.json) | ||
|
|
||
| Defines the host environments for testing: | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "OS": "ubuntu", | ||
| "RUNNER": "ubuntu-latest", | ||
| "TARGET": "x86_64-unknown-linux-gnu", | ||
| "run": ["always", "python", "node", "java"] | ||
| } | ||
| // ... other configurations | ||
| ] | ||
| ``` | ||
| * `OS`: The operating system of the host. | ||
| * `RUNNER`: The GitHub runner to use. | ||
| * `TARGET`: The target environment as defined in Rust. To see a list of available targets, run `rustup target list`. | ||
| * `run`: Specifies which language workflows should use this host configuration. The value `always` indicates that the configuration should be used for every workflow trigger. | ||
|
|
||
| #### Supported Languages Version (supported-languages-version.json) | ||
|
|
||
| Defines the supported versions of languages: | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "language": "java", | ||
| "versions": ["11", "17"], | ||
| "always-run-versions": ["17"] | ||
| }, | ||
| // ... other configurations | ||
| ] | ||
| ``` | ||
| * `language`: The language for which the version is supported. | ||
| * `versions`: The full versions supported of the language which will test against scheduled. | ||
| * `always-run-versions`: The versions that will always be tested, regardless of the workflow trigger. | ||
|
|
||
| #### Triggering Workflows | ||
|
|
||
| - Push to `main` by merging a PR or create a new pull request to run workflows automatically. | ||
| - Use `workflow_dispatch` for manual triggers, accepting a boolean configuration parameter to run all configurations. | ||
| - Scheduled runs are triggered daily to ensure regular testing of all configurations. | ||
|
|
||
| ### Mutual vs. Language-Specific Components | ||
|
|
||
| #### Mutual | ||
|
|
||
| - Matrix files - `.github/json_matrices/` | ||
| - Shared dependencies installation - `.github/workflows/install-shared-dependencies/action.yml` | ||
| - Rust linters - `.github/workflows/lint-rust/action.yml` | ||
|
|
||
| #### Language-Specific | ||
|
|
||
| - Package manager commands | ||
| - Testing frameworks | ||
| - Build processes | ||
|
|
||
| ### Customizing Workflows | ||
|
|
||
| Modify `<language>.yml` files to adjust language-specific steps. | ||
| Update matrix files to change tested versions or environments. | ||
| Adjust cron schedules in workflow files for different timing of scheduled runs. | ||
|
|
||
| ### Workflow Matrices | ||
|
|
||
| We use dynamic matrices for our CI/CD workflows, which are created using the `create-test-matrices` action. This action is defined in `.github/workflows/create-test-matrices/action.yml`. | ||
|
|
||
| #### How it works | ||
|
|
||
| 1. The action is called with a `language-name` input and `dispatch-run-full-matrix` input. | ||
| 2. It reads the `engine-matrix.json`, `build-matrix.json`, and `supported-languages-version.json` files. | ||
| 3. It filters the matrices based on the inputs and the event type. | ||
| 4. It generates three matrices: | ||
| - Engine matrix: Defines the types and versions of the engine to test against, for example Valkey 7.2.5. | ||
| - Host matrix: Defines the host platforms to run the tests on, for example Ubuntu on ARM64. | ||
| - Language-version matrix: Defines the supported versions of languages, for example python 3.8. | ||
|
|
||
| #### Outputs | ||
| - `engine-matrix-output`: The generated engine matrix. | ||
| - `host-matrix-output`: The generated host matrix. | ||
| - `language-version-matrix-output`: The generated language version matrix. | ||
|
|
||
| This dynamic matrix generation allows for flexible and efficient CI/CD workflows, adapting the test configurations based on the type of change and the specific language being tested. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,21 @@ | ||
| [ | ||
| { | ||
| "type": "valkey", | ||
| "version": "7.2.5" | ||
| }, | ||
| { | ||
| "type": "valkey", | ||
| "version": "8.0.0-rc1" | ||
| } | ||
| { | ||
| "type": "valkey", | ||
| "version": "8.0.0-rc1" | ||
| "run": "always" | ||
| }, | ||
| { | ||
| "type": "valkey", | ||
| "version": "7.2.5", | ||
| "run": "always" | ||
| }, | ||
| { | ||
| "type": "redis", | ||
| "version": "7.0" | ||
| }, | ||
| { | ||
| "type": "redis", | ||
| "version": "6.2", | ||
| "run": "always" | ||
| } | ||
| ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| [ | ||
| { | ||
| "language": "java", | ||
| "versions": ["11", "17"], | ||
| "always-run-versions": ["17"] | ||
| }, | ||
| { | ||
| "language": "python", | ||
| "versions": ["3.8", "3.9", "3.10", "3.11", "3.12"], | ||
| "always-run-versions": ["3.8", "3.12"] | ||
| }, | ||
| { | ||
| "language": "node", | ||
| "versions": ["16.x", "17.x", "18.x", "19.x", "20.x"], | ||
| "always-run-versions": ["16.x", "20.x"] | ||
| } | ||
| ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| inputs: | ||
| language-name: | ||
| description: "Language name" | ||
| required: true | ||
| type: choice | ||
| options: | ||
| - java | ||
| - node | ||
| - python | ||
| - go | ||
| - C# | ||
| run-full-matrix: | ||
| description: "Run the full matrix" | ||
| required: false | ||
| type: boolean | ||
| default: "false" | ||
|
|
||
| outputs: | ||
| engine-matrix-output: | ||
| description: "Engine matrix" | ||
| value: ${{ steps.load-engine-matrix.outputs.engine-matrix }} | ||
| host-matrix-output: | ||
| description: "Host matrix" | ||
| value: ${{ steps.load-host-matrix.outputs.host-matrix }} | ||
| version-matrix-output: | ||
| description: "Version matrix" | ||
| value: ${{ steps.create-lang-version-matrix.outputs.version-matrix }} | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Load engine matrix | ||
| id: load-engine-matrix | ||
| shell: bash | ||
| run: | | ||
| echo 'Select server engines to run tests against' | ||
| if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" || "${{ inputs.dispatch-run-full-matrix }}" == "false" ]]; then | ||
| echo 'Pick engines marked as `"run": "always"` only - on PR, push or manually triggered job which does not require full matrix' | ||
| echo engine-matrix=$(jq -c '[.[] | select(.run == "always")]' < .github/json_matrices/engine-matrix.json) >> $GITHUB_OUTPUT | ||
| else | ||
| echo 'Pick all engines - on cron (schedule) or if manually triggered job requires a full matrix' | ||
| echo engine-matrix=$(jq -c . < .github/json_matrices/engine-matrix.json) >> $GITHUB_OUTPUT | ||
| fi | ||
| echo "engine-matrix=$(cat $GITHUB_OUTPUT)" | ||
| - name: Load host matrix | ||
| id: load-host-matrix | ||
| shell: bash | ||
| run: | | ||
| echo 'Select runners (VMs) to run tests on' | ||
| if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" || "${{ inputs.dispatch-run-full-matrix }}" == "false" ]]; then | ||
| echo 'Pick runners marked as '"run": "always"' only - on PR, push or manually triggered job which does not require full matrix' | ||
| echo host-matrix=$(jq -c '[.[] | select(.run == "always")]' < .github/json_matrices/build-matrix.json) >> $GITHUB_OUTPUT | ||
| else | ||
| echo 'Pick all runners assigned for the chosen client (language) - on cron (schedule) or if manually triggered job requires a full matrix' | ||
| echo host-matrix=$(jq -c '[.[] | select(.languages | type == "array" and contains(["${{ inputs.language-name }}"]))]' < .github/json_matrices/build-matrix.json) >> $GITHUB_OUTPUT | ||
| fi | ||
| echo "host-matrix=$(cat $GITHUB_OUTPUT)" | ||
| - name: Create language version matrix | ||
| id: create-lang-version-matrix | ||
| shell: bash | ||
| run: | | ||
| echo 'Select language (framework/SDK) versions to run tests on' | ||
| if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" || "${{ inputs.dispatch-run-full-matrix }}" == "false" ]]; then | ||
| echo 'Pick language versions listed in 'always-run-versions' only - on PR, push or manually triggered job which does not require full matrix' | ||
| echo version-matrix=$(jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .["always-run-versions"] | map({version: .})]' < .github/json_matrices/supported-language-versions.json) >> $GITHUB_OUTPUT | ||
| else | ||
| echo 'Pick language versions listed in 'versions' - on cron (schedule) or if manually triggered job requires a full matrix' | ||
| echo version-matrix=$(jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .versions | map({version: .})]' < .github/json_matrices/supported-language-versions.json) >> $GITHUB_OUTPUT | ||
| fi | ||
| echo "version-matrix=$(cat $GITHUB_OUTPUT)" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.