From 743a70f05dec15f88541b563587593dc0036a880 Mon Sep 17 00:00:00 2001 From: Skisocks Date: Fri, 9 Jun 2023 11:17:04 +0100 Subject: [PATCH 1/3] docs(native-tekton): add initial docs explaining conversion procedure --- .../v3/admin/guides/native-tekton/_index.md | 14 ++++ .../native-tekton/pipeline-migration.md | 65 +++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 content/en/v3/admin/guides/native-tekton/_index.md create mode 100644 content/en/v3/admin/guides/native-tekton/pipeline-migration.md diff --git a/content/en/v3/admin/guides/native-tekton/_index.md b/content/en/v3/admin/guides/native-tekton/_index.md new file mode 100644 index 00000000000..5b2f0785ece --- /dev/null +++ b/content/en/v3/admin/guides/native-tekton/_index.md @@ -0,0 +1,14 @@ +--- +title: Native Tekton +linktitle: Native Tekton +type: docs +description: How to integrate existing clusters with native Tekton +weight: 100 +aliases: + - /v3/guides/native-tekton +--- + +As part of the efforts to upgrade Jenkins X's compatibility with more recent Kubernetes & Tekton versions we've made the +decision to fully migrate to native Tekton pipelines. This brings us inline with the latest version of Tekton; +and allows us to take advantage of features such as the Tekton Dashboard and whilst increasing the transparency of the +pipeline mechanisms. diff --git a/content/en/v3/admin/guides/native-tekton/pipeline-migration.md b/content/en/v3/admin/guides/native-tekton/pipeline-migration.md new file mode 100644 index 00000000000..cdca104f0dc --- /dev/null +++ b/content/en/v3/admin/guides/native-tekton/pipeline-migration.md @@ -0,0 +1,65 @@ +--- +title: Pipeline Migration +linktitle: Pipeline Migration +type: docs +description: How to migrate existing pipelines to native Tekton +weight: 100 +aliases: + - /v3/guides/native-tekton +--- + +Migrating to native Tekton brings a number of changes to the way that Jenkins X pipelines are configured and run. +The main changes are: +- The lighthouse PipelineRun resolver (responsible for handling the `uses:sourceURI` syntax) has been replaced with +[Tekton's git resolver](https://tekton.dev/docs/pipelines/git-resolver/). +- Pipelines are no longer confined to a single pod running multiple containers (1 task, many steps). Instead, each step +now runs as its own pod (many tasks, many steps). +- As each pipeline now contains many tasks, [Tekton's workspaces](https://tekton.dev/docs/pipelines/workspaces/) are used to share files between tasks. + +This guide will walk you through the steps required to migrate your existing pipelines to native Tekton. + +## Prerequisites +You're cluster git repository will need to be at `versionStream>=[insert version here]` before proceeding with the +pipeline upgrade. This can be done by running the following command on your cluster git repository. +```bash +jx gitops upgrade +``` +The upgrade will bring in the relevant pipeline changes for your cluster repository so the `verify` & `bootjob` pipelines +will continue to work during your migration. + +## Upgrading pipelines in your repositories +You can upgrade the pipelines in all the repositories you've imported or created via quickstarts by running: +```bash +jx updatebot pipeline +``` +This uses the default [kpt](https://kpt.dev/) strategy of `resource-merge` to merge any local changes with the those in the +[pipeline-catalog](https://github.com/jenkins-x/jx3-pipeline-catalog). The strategy used can be changed using the `--strategy` flag. +For more information see`jx updatebot pipeline --help`. + +**Warning:** This is substantial change to the pipeline configuration and will likely result in merge conflicts. +If so, you will need to resolve these manually or by using the pipeline conversion tool seen in [converting pipelines](#converting-pipelines). + +## Converting pipelines +Both in-repository pipelines and personal catalogs can be converted by using the following command in the root of the +repository: +```bash +jx pipeline convert remotetasks +``` + +The command identifies whether the PipelineRun is child or parent (in-repository or catalog) by checking for the presence +of a `uses:sourceURI` image in the stepTemplate. + +Child PipelineRuns have they're steps converted to either taskRefs or embeddedTasks depending on whether the original step was inherited. + +Parent PipelineRuns have their steps converted into standalone tasks as native Tekton does not support PipelineRuns +inheriting tasks from other PipelineRuns. + +Standalone Tasks have the Lighthouse's default params and environment variables added to them. These can be found + [here](https://jenkins-x.io/v3/develop/reference/variables/#environment-variables). + +By default, the workspace size of the converted pipeline is calculated from the size of the repository plus 300Mi - to account +for any additional files that may be added during the pipeline run. This can be overridden using the `--workspace-size` flag. +For catalogs the calculation can be disabled by setting `--calculate-workspace-volume=false` in which case the workspace size +will default to 1Gi. + +For more information on this command, see `jx pipeline convert remotetasks --help`. From 9d8b90d665bc94368070d59542a8748370e80430 Mon Sep 17 00:00:00 2001 From: Skisocks Date: Fri, 9 Jun 2023 13:11:10 +0100 Subject: [PATCH 2/3] fix(pipeline-convert): replace existing pipeline convert -> pipeline convert uses --- content/en/v3/about/changes.md | 2 +- content/en/v3/admin/guides/migrate/v2.md | 4 ++-- content/en/v3/admin/guides/migrate/v3-alpha.md | 4 ++-- content/en/v3/develop/pipelines/upgrading.md | 6 +++--- .../en/v3/develop/reference/jx/pipeline/convert/_index.md | 8 ++++---- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/content/en/v3/about/changes.md b/content/en/v3/about/changes.md index 69ec97fc89e..3cfaee84c6d 100644 --- a/content/en/v3/about/changes.md +++ b/content/en/v3/about/changes.md @@ -41,7 +41,7 @@ updatebot fatal: could not read Username for 'https://github.com': No such devic * its now much easier to [write system tests against Preview Environments](https://github.com/jenkins-x/jx-preview#system-tests-in-previews) so it's easier to test images and charts function as you expect inside a Pull Request before you are happy to merge the work for faster feedback * check out the new [DevOps, GitOps and Cloud Native](https://jenkins-x.io/v3/devops/) documentation we're putting together based on the learnings of continuously deliverying Jenkins X with Jenkins X. * new clusters created using Terraform that use Vault will be using the `jx-vault` namespace to setup Vault (so that its managed by Terraform) -* you can now use [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) to [convert any old pipelines](/v3/develop/pipelines/upgrading/#converting-older-pipelines) across to the latest [concise syntax](/v3/develop/pipelines/catalog/) +* you can now use [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) to [convert any old pipelines](/v3/develop/pipelines/upgrading/#converting-older-pipelines) across to the latest [concise syntax](/v3/develop/pipelines/catalog/) * we have an awesome [new syntax to help share pipelines across git repositories](/v3/develop/pipelines/catalog/) that makes it easier to simplify the pipelines in each repository while keeping things vanilla Tekton YAML and letting you override and customise anything anywhere * the boot job now upgrades the `docs` folder to show what charts and versions are deployed in each namespace. You can view the `docs` folder in your own git repositories once you've [upgraded your cluster](/v3/admin/setup/upgrades/cluster/) * you can see the default reports for [kubernetes](https://github.com/jx3-gitops-repositories/jx3-kubernetes/tree/master/docs ), [aws](https://github.com/jx3-gitops-repositories/jx3-eks-vault/tree/master/docs), [azure](https://github.com/jx3-gitops-repositories/jx3-azure-akv) and [gke](https://github.com/jx3-gitops-repositories/jx3-gke-gsm/tree/master/docs ) diff --git a/content/en/v3/admin/guides/migrate/v2.md b/content/en/v3/admin/guides/migrate/v2.md index 15e6082cae0..78b8f94f128 100644 --- a/content/en/v3/admin/guides/migrate/v2.md +++ b/content/en/v3/admin/guides/migrate/v2.md @@ -19,10 +19,10 @@ You can then test the projects work fine in the new v3 cluster. If they do you c ## Converting pipelines -You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository: +You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository: ```bash -jx pipeline convert +jx pipeline convert uses ``` ## Batch importing diff --git a/content/en/v3/admin/guides/migrate/v3-alpha.md b/content/en/v3/admin/guides/migrate/v3-alpha.md index 3dd916ebfa5..884d1674ec6 100644 --- a/content/en/v3/admin/guides/migrate/v3-alpha.md +++ b/content/en/v3/admin/guides/migrate/v3-alpha.md @@ -28,10 +28,10 @@ You can use the `--strategy` to change this default if you wish. See `jx updateb ## Converting pipelines -You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository: +You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository: ```bash -jx pipeline convert +jx pipeline convert uses ``` ### If you lose some pipeline changes diff --git a/content/en/v3/develop/pipelines/upgrading.md b/content/en/v3/develop/pipelines/upgrading.md index 90cdd50adb1..b61fbf14a37 100644 --- a/content/en/v3/develop/pipelines/upgrading.md +++ b/content/en/v3/develop/pipelines/upgrading.md @@ -9,10 +9,10 @@ weight: 500 ## Converting older pipelines -If you have older pipelines in your git repository created with an older version of Jenkins X you can convert them to the new concise `image: uses:sourceURI` syntax via the [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command: +If you have older pipelines in your git repository created with an older version of Jenkins X you can convert them to the new concise `image: uses:sourceURI` syntax via the [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command: ```bash -jx pipeline convert +jx pipeline convert uses ``` The pipelines should be modified and if you have a `Kptfile` it will be removed. @@ -76,4 +76,4 @@ You can now perform a batch of Pull Requests if you need to upgrade your pipelin See: [generate pull requests to upgrade pipelines](/v3/admin/guides/migrate/v3-alpha/#upgrading-pipelines-in-your-repositories) -Here is an example of [an automated Pull Request](https://github.com/jenkins-x/jx-gitops/pull/551) that was used to migrate one of our pipelines on [jx-gitops](https://github.com/jenkins-x/jx-gitops) - as you can see the resulting file is much simpler and easier to maintain. \ No newline at end of file +Here is an example of [an automated Pull Request](https://github.com/jenkins-x/jx-gitops/pull/551) that was used to migrate one of our pipelines on [jx-gitops](https://github.com/jenkins-x/jx-gitops) - as you can see the resulting file is much simpler and easier to maintain. diff --git a/content/en/v3/develop/reference/jx/pipeline/convert/_index.md b/content/en/v3/develop/reference/jx/pipeline/convert/_index.md index 5a9d5f38d0b..220c9f3f170 100644 --- a/content/en/v3/develop/reference/jx/pipeline/convert/_index.md +++ b/content/en/v3/develop/reference/jx/pipeline/convert/_index.md @@ -1,5 +1,5 @@ --- -title: jx pipeline convert +title: jx pipeline convert uses linktitle: convert type: docs description: "Converts the pipelines to use the 'image: uses:sourceURI' include mechanism" @@ -10,7 +10,7 @@ aliases: ### Usage ``` -jx pipeline convert +jx pipeline convert uses ``` ### Synopsis @@ -24,10 +24,10 @@ So that pipelines are smaller, simpler and easier to upgrade pipelines with the ```bash # Convert a repository created using the alpha/beta of v3 # to use the nice new uses: syntax - jx pipeline convert + jx pipeline convert uses # Convert a pipeline catalog to the uses syntax and layout - jx pipeline convert --catalog + jx pipeline convert uses --catalog ``` ### Options From baf671e039f1a677b4faeb77a8df0e8ed0214eaa Mon Sep 17 00:00:00 2001 From: Skisocks Date: Wed, 14 Jun 2023 13:37:47 +0100 Subject: [PATCH 3/3] docs(native-tekton): add Changes to Pipelines section --- .../native-tekton/changes-to-pipelines.md | 100 ++++++++++++++++++ .../native-tekton/pipeline-migration.md | 10 -- 2 files changed, 100 insertions(+), 10 deletions(-) create mode 100644 content/en/v3/admin/guides/native-tekton/changes-to-pipelines.md diff --git a/content/en/v3/admin/guides/native-tekton/changes-to-pipelines.md b/content/en/v3/admin/guides/native-tekton/changes-to-pipelines.md new file mode 100644 index 00000000000..3851598b536 --- /dev/null +++ b/content/en/v3/admin/guides/native-tekton/changes-to-pipelines.md @@ -0,0 +1,100 @@ +--- +title: Changes to Pipelines +linktitle: Changes to Pipelines +type: docs +description: Describes how the pipelines have changed with the move to native Tekton +weight: 100 +aliases: + - /v3/guides/native-tekton +--- + +Migrating to native Tekton brings a number of changes to the way that Jenkins X pipelines are configured and run. +The main changes are: +- The lighthouse PipelineRun resolver (responsible for handling the `uses:sourceURI` syntax) has been replaced with +[Tekton's git resolver](https://tekton.dev/docs/pipelines/git-resolver/). +- Pipelines are no longer confined to a single pod running multiple containers (1 task, many steps). Instead, each step +now runs as its own pod (many tasks, many steps). This is to allow for the use of taskRefs to inherit tasks. Tekton +doesn't currently allow for step inheritance. +- As each pipeline now contains many tasks, [Tekton's workspaces](https://tekton.dev/docs/pipelines/workspaces/) are +used to share files between tasks. + +For more information regarding how the new pipelines are configured, please see the [Tekton Docs](https://tekton.dev/docs/). + +## Functionality Changes +We've tried to keep the functionality as close to the original as possible, however there are a few changes +that are worth noting. + +### Version Stream +The version stream functionality continues to exist as before, so any pipelines using the +[pipeline catalog](https://github.com/jenkins-x/jx3-pipeline-catalog) will continue to have their version resolved by +lighthouse and the `versionStream` defined in your cluster repository. + +Example git resolver version stream usage: +```yaml +params: + - name: url + value: https://github.com/spring-financial-group/jx3-pipeline-catalog.git + - name: revision + value: versionStream + - name: pathInRepo + value: tasks/git-clone/git-clone.yaml +``` + +### Lighthouse Default Params +Lighthouse will continue to add and populate the [default params and environment variables](https://jenkins-x.io/v3/develop/reference/variables/#environment-variables) +on PipelineRuns as before. However, due to taskRef resolution now being handled by Tekton, any standalone task will need +to have these explicitly defined. Propagated params and workspaces are part of the [Tekton roadmap](https://github.com/tektoncd/community/blob/main/roadmap.md#roadmap) +so will hopefully be available in the future. + +### Overwriting Inherited Tasks +Previously it was possible to overwrite certain fields of inherited steps/tasks by redefining them in the child PipelineRun. +Unfortunately, Tekton does not currently support this functionality. As a result, to overwrite an inherited task, you +will need to define an embedded task on the child PipelineRun and copy in the relevant fields. + +Inherited Task: +```yaml +- name: build-make-linux + runAfter: + - jx-variables + taskRef: + params: + - name: url + value: https://github.com/jenkins-x/jx3-pipeline-catalog.git + - name: revision + value: versionStream + - name: pathInRepo + value: tasks/go/pullrequest/build-make-linux.yaml + resolver: git + workspaces: + - name: output + workspace: pipeline-ws +``` + +Embedded Task: +```yaml +- name: build-make-linux + runAfter: + - jx-variables + taskSpec: + stepTemplate: + workingDir: /workspace/source + steps: + - image: golang:1.19.3@sha256:7ffa70183b7596e6bc1b78c132dbba9a6e05a26cd30eaa9832fecad64b83f029 + name: build-make-linux + resources: {} + script: | + #!/bin/sh + make linux + workspaces: + - description: The workspace used to store the cloned git repository and the generated + files + mountPath: /workspace + name: output + workspaces: + - name: output + workspace: pipeline-ws +``` + +### Pipeline Catalog +The structure of the pipeline catalog has changed slightly. The `/tasks` directory will now only contain the tasks +themselves, with the PipelineRun definitions now only residing in the `/packs` directory. diff --git a/content/en/v3/admin/guides/native-tekton/pipeline-migration.md b/content/en/v3/admin/guides/native-tekton/pipeline-migration.md index cdca104f0dc..71168cbfac1 100644 --- a/content/en/v3/admin/guides/native-tekton/pipeline-migration.md +++ b/content/en/v3/admin/guides/native-tekton/pipeline-migration.md @@ -8,16 +8,6 @@ aliases: - /v3/guides/native-tekton --- -Migrating to native Tekton brings a number of changes to the way that Jenkins X pipelines are configured and run. -The main changes are: -- The lighthouse PipelineRun resolver (responsible for handling the `uses:sourceURI` syntax) has been replaced with -[Tekton's git resolver](https://tekton.dev/docs/pipelines/git-resolver/). -- Pipelines are no longer confined to a single pod running multiple containers (1 task, many steps). Instead, each step -now runs as its own pod (many tasks, many steps). -- As each pipeline now contains many tasks, [Tekton's workspaces](https://tekton.dev/docs/pipelines/workspaces/) are used to share files between tasks. - -This guide will walk you through the steps required to migrate your existing pipelines to native Tekton. - ## Prerequisites You're cluster git repository will need to be at `versionStream>=[insert version here]` before proceeding with the pipeline upgrade. This can be done by running the following command on your cluster git repository.