Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEP074] Remove Storage, Git and Generic PipelineResources #6150

Merged
merged 1 commit into from
Mar 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions cmd/controller/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,8 @@ func main() {
flag.StringVar(&opts.Images.EntrypointImage, "entrypoint-image", "", "The container image containing our entrypoint binary.")
flag.StringVar(&opts.Images.SidecarLogResultsImage, "sidecarlogresults-image", "", "The container image containing the binary for accessing results.")
flag.StringVar(&opts.Images.NopImage, "nop-image", "", "The container image used to stop sidecars")
flag.StringVar(&opts.Images.GitImage, "git-image", "", "The container image containing our Git binary.")
flag.StringVar(&opts.Images.ShellImage, "shell-image", "", "The container image containing a shell")
flag.StringVar(&opts.Images.ShellImageWin, "shell-image-win", "", "The container image containing a windows shell")
flag.StringVar(&opts.Images.GsutilImage, "gsutil-image", "", "The container image containing gsutil")
flag.StringVar(&opts.Images.WorkingDirInitImage, "workingdirinit-image", "", "The container image containing our working dir init binary.")

// This parses flags.
Expand Down
2 changes: 0 additions & 2 deletions cmd/webhook/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ import (
"github.com/tektoncd/pipeline/pkg/apis/resolution"
resolutionv1alpha1 "github.com/tektoncd/pipeline/pkg/apis/resolution/v1alpha1"
resolutionv1beta1 "github.com/tektoncd/pipeline/pkg/apis/resolution/v1beta1"
resourcev1alpha1 "github.com/tektoncd/pipeline/pkg/apis/resource/v1alpha1"
"k8s.io/apimachinery/pkg/runtime/schema"
"knative.dev/pkg/configmap"
"knative.dev/pkg/controller"
Expand All @@ -52,7 +51,6 @@ import (

var types = map[schema.GroupVersionKind]resourcesemantics.GenericCRD{
// v1alpha1
v1alpha1.SchemeGroupVersion.WithKind("PipelineResource"): &resourcev1alpha1.PipelineResource{},
v1alpha1.SchemeGroupVersion.WithKind("Run"): &v1alpha1.Run{},
v1alpha1.SchemeGroupVersion.WithKind("VerificationPolicy"): &v1alpha1.VerificationPolicy{},
// v1beta1
Expand Down
5 changes: 2 additions & 3 deletions config/200-clusterrole.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ rules:
# Controller needs cluster access to all of the CRDs that it is responsible for
# managing.
- apiGroups: ["tekton.dev"]
resources: ["tasks", "clustertasks", "taskruns", "pipelines", "pipelineruns", "pipelineresources", "runs", "customruns"]
resources: ["tasks", "clustertasks", "taskruns", "pipelines", "pipelineruns", "runs", "customruns"]
verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
- apiGroups: ["tekton.dev"]
resources: ["verificationpolicies"]
Expand All @@ -37,7 +37,7 @@ rules:
resources: ["taskruns/finalizers", "pipelineruns/finalizers", "runs/finalizers", "customruns/finalizers"]
verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
- apiGroups: ["tekton.dev"]
resources: ["tasks/status", "clustertasks/status", "taskruns/status", "pipelines/status", "pipelineruns/status", "pipelineresources/status", "runs/status", "customruns/status", "verificationpolicies/status"]
resources: ["tasks/status", "clustertasks/status", "taskruns/status", "pipelines/status", "pipelineruns/status", "runs/status", "customruns/status", "verificationpolicies/status"]
verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
# resolution.tekton.dev
- apiGroups: ["resolution.tekton.dev"]
Expand Down Expand Up @@ -92,7 +92,6 @@ rules:
- tasks.tekton.dev
- clustertasks.tekton.dev
- taskruns.tekton.dev
- pipelineresources.tekton.dev
- resolutionrequests.resolution.tekton.dev
- customruns.tekton.dev
- verificationpolicies.tekton.dev
Expand Down
52 changes: 0 additions & 52 deletions config/300-resource.yaml

This file was deleted.

1 change: 0 additions & 1 deletion config/clusterrole-aggregate-edit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ rules:
- taskruns
- pipelines
- pipelineruns
- pipelineresources
- runs
- customruns
verbs:
Expand Down
1 change: 0 additions & 1 deletion config/clusterrole-aggregate-view.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ rules:
- taskruns
- pipelines
- pipelineruns
- pipelineresources
- runs
- customruns
verbs:
Expand Down
33 changes: 0 additions & 33 deletions config/config-artifact-bucket.yaml

This file was deleted.

28 changes: 0 additions & 28 deletions config/config-artifact-pvc.yaml

This file was deleted.

4 changes: 0 additions & 4 deletions config/controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,11 @@ spec:
args: [
# These images are built on-demand by `ko resolve` and are replaced
# by image references by digest.
"-git-image", "ko://github.com/tektoncd/pipeline/cmd/git-init",
"-entrypoint-image", "ko://github.com/tektoncd/pipeline/cmd/entrypoint",
"-nop-image", "ko://github.com/tektoncd/pipeline/cmd/nop",
"-sidecarlogresults-image", "ko://github.com/tektoncd/pipeline/cmd/sidecarlogresults",
"-workingdirinit-image", "ko://github.com/tektoncd/pipeline/cmd/workingdirinit",

# This is gcr.io/google.com/cloudsdktool/cloud-sdk:302.0.0-slim
JeromeJu marked this conversation as resolved.
Show resolved Hide resolved
"-gsutil-image", "gcr.io/google.com/cloudsdktool/cloud-sdk@sha256:27b2c22bf259d9bc1a291e99c63791ba0c27a04d2db0a43241ba0f1f20f4067f",
JeromeJu marked this conversation as resolved.
Show resolved Hide resolved

# The shell image must allow root in order to create directories and copy files to PVCs.
# cgr.dev/chainguard/busybox as of April 14 2022
# image shall not contains tag, so it will be supported on a runtime like cri-o
Expand Down
123 changes: 0 additions & 123 deletions docs/additional-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,6 @@ installation.

## Table of Contents

- [Configuring PipelineResource storage](#configuring-pipelineresource-storage)
- [Configuring a persistent volume](#configuring-a-persistent-volume)
- [Configuring a cloud storage bucket](#configuring-a-cloud-storage-bucket)
- [Example configuration for an S3 bucket](#example-configuration-for-an-s3-bucket)
- [Example configuration for a GCS bucket](#example-configuration-for-a-gcs-bucket)
- [Configuring built-in remote Task and Pipeline resolution](#configuring-built-in-remote-task-and-pipeline-resolution)
- [Configuring CloudEvents notifications](#configuring-cloudevents-notifications)
- [Configuring self-signed cert for private registry](#configuring-self-signed-cert-for-private-registry)
Expand All @@ -37,124 +32,6 @@ installation.
- [Verify Tekton Resources](#verify-tekton-resources)
- [Next steps](#next-steps)

## Configuring PipelineResource storage

> :warning: **`PipelineResources` are [deprecated](deprecations.md#deprecation-table).**
>
> For storage, consider using [`Workspaces`](workspaces.md) with [`VolumeClaimTemplates`](https://github.com/tektoncd/pipeline/blob/main/docs/workspaces.md#volumeclaimtemplate)
> to automatically provision and manage Persistent Volume Claims (PVCs). Read more in [TEP-0074](https://github.com/tektoncd/community/blob/main/teps/0074-deprecate-pipelineresources.md).

PipelineResources are one of the ways that Tekton passes data between Tasks. If you intend to
use PipelineResources in your Pipelines then you'll need to configure a storage location
for that data to be put so that it can be shared between Tasks in the Pipeline.

The storage options available for sharing PipelineResources between Tasks in a Pipeline are:

* [A persistent volume](#configuring-a-persistent-volume)
* [A cloud storage bucket](#configuring-a-cloud-storage-bucket)

Either option provides the same functionality to Tekton Pipelines. Choose the option that
best suits your business needs. For example:

- In some environments, creating a persistent volume could be slower than transferring files to/from a cloud storage bucket.
- If the cluster is running in multiple zones, accessing a persistent volume could be unreliable.

**Note:** To customize the names of the `ConfigMaps` for artifact persistence (e.g. to avoid collisions with other services), rename the `ConfigMap` and update the env value defined [controller.yaml](https://github.com/tektoncd/pipeline/blob/e153c6f2436130e95f6e814b4a792fb2599c57ef/config/controller.yaml#L66-L75).

### Configuring a persistent volume

To configure a [persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), use a `ConfigMap` with the name `config-artifact-pvc` and the following attributes:

- `size`: the size of the volume. Default is 5GiB.
- `storageClassName`: the [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) of the volume. The possible values depend on the cluster configuration and the underlying infrastructure provider. Default is the default storage class.

### Configuring a cloud storage bucket

To configure either an [S3 bucket](https://aws.amazon.com/s3/) or a [GCS bucket](https://cloud.google.com/storage/),
use a `ConfigMap` with the name `config-artifact-bucket` and the following attributes:

- `location` - the address of the bucket, for example `gs://mybucket` or `s3://mybucket`.
- `bucket.service.account.secret.name` - the name of the secret containing the credentials for the service account with access to the bucket.
- `bucket.service.account.secret.key` - the key in the secret with the required
service account JSON file.
- `bucket.service.account.field.name` - the name of the environment variable to use when specifying the
secret path. Defaults to `GOOGLE_APPLICATION_CREDENTIALS`. Set to `BOTO_CONFIG` if using S3 instead of GCS.

**Important:** Configure your bucket's retention policy to delete all files after your `Tasks` finish running.

**Note:** You can only use an S3 bucket located in the `us-east-1` region. This is a limitation of [`gsutil`](https://cloud.google.com/storage/docs/gsutil) running a `boto` configuration behind the scenes to access the S3 bucket.
afrittoli marked this conversation as resolved.
Show resolved Hide resolved


#### Example configuration for an S3 bucket

Below is an example configuration that uses an S3 bucket:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: tekton-storage
namespace: tekton-pipelines
type: kubernetes.io/opaque
stringData:
boto-config: |
[Credentials]
aws_access_key_id = AWS_ACCESS_KEY_ID
aws_secret_access_key = AWS_SECRET_ACCESS_KEY
[s3]
host = s3.us-east-1.amazonaws.com
[Boto]
https_validate_certificates = True
---
apiVersion: v1
kind: ConfigMap
metadata:
name: config-artifact-bucket
namespace: tekton-pipelines
data:
location: s3://mybucket
bucket.service.account.secret.name: tekton-storage
bucket.service.account.secret.key: boto-config
bucket.service.account.field.name: BOTO_CONFIG
```

#### Example configuration for a GCS bucket

Below is an example configuration that uses a GCS bucket:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: tekton-storage
namespace: tekton-pipelines
type: kubernetes.io/opaque
stringData:
gcs-config: |
{
"type": "service_account",
"project_id": "gproject",
"private_key_id": "some-key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nME[...]dF=\n-----END PRIVATE KEY-----\n",
"client_email": "tekton-storage@gproject.iam.gserviceaccount.com",
"client_id": "1234567890",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/tekton-storage%40gproject.iam.gserviceaccount.com"
}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: config-artifact-bucket
namespace: tekton-pipelines
data:
location: gs://mybucket
bucket.service.account.secret.name: tekton-storage
bucket.service.account.secret.key: gcs-config
bucket.service.account.field.name: GOOGLE_APPLICATION_CREDENTIALS
```

## Configuring built-in remote Task and Pipeline resolution

Expand Down
Loading