Skip to content

Commit

Permalink
docs: Update local-test-setup.md to use scripts (kyma-project#2178)
Browse files Browse the repository at this point in the history 
* docs: Update local-test-setup.md to use scripts

* Update docs/contributor/04-local-test-setup.md

* docs: Reference Mandatory Modules Controller in Architecture Docs (kyma-project#2197)

* reference controller in architecture docs

* fix markdown lint

* improve docs

* feat: Implement MaintenanceWindow determination logic (kyma-project#2196)

* feat: Add metadata and status helpers to Kyma type

* add go sum

* cleanup go mod, add api to coverage

* remove api from unit-test coverage

* feat: MaintenanceWindow service

* chore(dependabot): bump k8s.io/apimachinery from 0.32.0 to 0.32.1 in /api (kyma-project#2185)

chore(dependabot): bump k8s.io/apimachinery in /api

Bumps [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) from 0.32.0 to 0.32.1.
- [Commits](kubernetes/apimachinery@v0.32.0...v0.32.1)

---
updated-dependencies:
- dependency-name: k8s.io/apimachinery
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(dependabot): bump sigs.k8s.io/controller-runtime from 0.19.4 to 0.20.0 (kyma-project#2192)

chore(dependabot): bump sigs.k8s.io/controller-runtime

Bumps [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) from 0.19.4 to 0.20.0.
- [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases)
- [Changelog](https://github.com/kubernetes-sigs/controller-runtime/blob/main/RELEASE.md)
- [Commits](kubernetes-sigs/controller-runtime@v0.19.4...v0.20.0)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/controller-runtime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: Add metadata and status helpers to Kyma type

* cleanup go mod, add api to coverage

* remove obsolete comment

* fix fake arguments in suite_test

* avoid handler name in suite_test

* rename to maintenanceWindow consistently

* underscore unused receiver arg

* omit receiver arg

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* title numbering and casing

* Apply suggestions from code review

Co-authored-by: Małgorzata Świeca <malgorzata.swieca@sap.com>

* Update docs/contributor/04-local-test-setup.md

* indent scripts

* link CRDs

* Update docs/contributor/04-local-test-setup.md

Co-authored-by: Małgorzata Świeca <malgorzata.swieca@sap.com>

* Update docs/contributor/04-local-test-setup.md

Co-authored-by: Małgorzata Świeca <malgorzata.swieca@sap.com>

* fix indentation

* fix indentation and numbering

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Amritanshu Sikdar <amritanshu.sikdar@sap.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Lindner <50365642+lindnerby@users.noreply.github.com>
Co-authored-by: Małgorzata Świeca <malgorzata.swieca@sap.com>
  • Loading branch information
@c-pius @amritanshusikdar
@dependabot @lindnerby @mmitoraj
5 people authored Jan 22, 2025
1 parent 4530763 commit 1fd218e
Show file tree
Hide file tree
Showing 3 changed files with 120 additions and 240 deletions.
332 changes: 104 additions & 228 deletions docs/contributor/04-local-test-setup.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
# Local Test Setup in the Control Plane Mode Using k3d

> ### Supported Versions
> For more information on the tooling versions expected in the project, see [`versions.yaml`](../../versions.yaml).
# Configure a Local Test Setup

## Context

Expand All @@ -16,283 +13,162 @@ This setup is deployed with the following security features enabled:
* Strict mTLS connection between Kyma Control Plane (KCP) and SKR clusters
* SAN Pinning (SAN of client TLS certificate needs to match the DNS annotation of a corresponding Kyma CR)

## Procedure

### KCP Cluster Setup

1. Create a local KCP cluster:

```shell
k3d cluster create kcp-local --port 9443:443@loadbalancer \
--registry-create k3d-registry.localhost:0.0.0.0:5111 \
--k3s-arg '--disable=traefik@server:0' \
--k3s-arg --tls-san=host.k3d.internal@server:*
```
## Prerequisites

2. Open `/etc/hosts` file on your local system:
Install the following tooling in the versions defined in [`versions.yaml`](../../versions.yaml):

```shell
sudo nano /etc/hosts
```
- [Docker](https://www.docker.com/)
- [Go](https://go.dev/)
- [golangci-lint](https://golangci-lint.run/)
- [istioctl](https://istio.io/latest/docs/ops/diagnostic-tools/istioctl/)
- [k3d](https://k3d.io/stable/)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [kustomize](https://kustomize.io/)
- [modulectl](https://github.com/kyma-project/modulectl)
- [yq](https://github.com/mikefarah/yq/tree/master)

Add an entry for your local k3d registry created in step 1:
## Procedure

```txt
127.0.0.1 k3d-registry.localhost
```
Follow the steps using scripts from the project root.

3. Install the following prerequisites required by Lifecycle Manager:
### 1. Create Test Clusters

1. Istio CRDs using `istioctl`:
Create local test clusters for SKR and KCP.

```shell
brew install istioctl && \
istioctl install --set profile=demo -y
```
```sh
K8S_VERSION=$(yq e '.k8s' ./versions.yaml)
CERT_MANAGER_VERSION=$(yq e '.certManager' ./versions.yaml)
./scripts/tests/create_test_clusters.sh --k8s-version $K8S_VERSION --cert-manager-version $CERT_MANAGER_VERSION
```

2. `cert-manager` by Jetstack:
### 2. Install the Custom Resource Definitions

```shell
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml
```
Install the [Lifecycle Manager CRDs](./resources/README.md) in the KCP cluster.

4. Deploy Lifecycle Manager in the cluster:
```sh
./scripts/tests/install_crds.sh
```

```shell
make local-deploy-with-watcher IMG=europe-docker.pkg.dev/kyma-project/prod/lifecycle-manager:latest
```
### 3. Deploy Lifecycle Manager

> **TIP:** If you get an error similar to the following, wait a couple of seconds and rerun the command.
>
> ```shell
> Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
> ```
You can deploy Lifecycle Manager either from the registry or local sources. Choose one of the below options:

<details>
<summary>Custom Lifecycle Manager image deployment</summary>
If you want to test a custom image of Lifecycle Manager, run the following:
1. Deploy a built image from the registry, for example, the `latest` image from the `prod` registry.

```shell
# build your local image
make docker-build IMG=k3d-registry.localhost:5111/<your-image-name>:<your-image-tag>
# push the image to the local registry
make docker-push IMG=k3d-registry.localhost:5111/<your-image-name>:<your-image-tag>
# deploy Lifecycle Manager using the image (note the change to port 5000 which is exposed in the cluster)
make local-deploy-with-watcher IMG=k3d-registry.localhost:5000/<your-image-name>:<your-image-tag>
```

</details>
```sh
REGISTRY=prod
TAG=latest
./scripts/tests/deploy_klm_from_registry.sh --image-registry $REGISTRY --image-tag $TAG
```

5. Create a ModuleTemplate CR using [modulectl](https://github.com/kyma-project/modulectl).
The ModuleTemplate CR includes component descriptors for module installations.
1. Build a new image from the local sources, push it to the local KCP registry, and deploy it.

In this tutorial, we will create a ModuleTemplate CR from the [`template-operator`](https://github.com/kyma-project/template-operator) repository.
Adjust the path to your `template-operator` local directory or any other reference module operator accordingly.

```shell
cd <local path to template operator repository>
# generate the manifests and save them to the template-operator.yaml file
make build-manifests
# create the a ModuleTemplate CR and save it to the template.yaml file
modulectl create --config-file ./module-config.yaml --registry http://k3d-registry.localhost:5111 --insecure
```
```sh
./scripts/tests/deploy_klm_from_sources.sh
```

6. Verify images pushed to the local registry:
### 4. Deploy a Kyma CR

```shell
curl http://k3d-registry.localhost:5111/v2/_catalog\?n\=100
```
```sh
SKR_HOST=host.k3d.internal
./scripts/tests/deploy_kyma.sh $SKR_HOST
```

The output should look like the following:
### 5. Verify If the Kyma CR Becomes Ready

```shell
{"repositories":["component-descriptors/kyma-project.io/module/template-operator"]}
```
1. Verify if the Kyma CR is in the `Ready` state in KCP. It takes roughly 1–2 minutes.

7. Open the generated `template.yaml` file and change the following line:
```sh
kubectl config use-context k3d-kcp
kubectl get kyma/kyma-sample -n kcp-system
```

```yaml
<...>
- baseUrl: k3d-registry.localhost:5111
<...>
```

To the following:
1. Verify if the Kyma CR is in the `Ready` state in SKR. It takes roughly 1-2 minutes.

```yaml
<...>
- baseUrl: k3d-registry.localhost:5000
<...>
```
```sh
kubectl config use-context k3d-skr
kubectl get kyma/default -n kyma-system
```

You need the change because the operators are running inside of two local k3d clusters, and the internal port for the k3d registry is set by default to `5000`.
### 6. [OPTIONAL] Deploy the template-operator Module

8. Apply the template:
Build the template-operator module from the local sources, push it to the local KCP registry, and deploy it.

```shell
kubectl apply -f ./template.yaml
```
```sh
cd <template-operator-repository>
### SKR Cluster Setup
make build-manifests
modulectl create --config-file ./module-config.yaml --registry http://localhost:5111 --insecure
Create a local Kyma runtime (SKR) cluster:
kubectl config use-context k3d-kcp
# repository URL is localhost:5111 on the host machine but must be k3d-kcp-registry.localhost:5000 within the cluster
yq e '.spec.descriptor.component.repositoryContexts[0].baseUrl = "k3d-kcp-registry.localhost:5000"' ./template.yaml | kubectl apply -f -
```shell
k3d cluster create skr-local --k3s-arg --tls-san=host.k3d.internal@server:*
```
MT_VERSION=$(yq e '.spec.version' ./template.yaml)
cd <lifecycle-manager-repository>
./scripts/tests/deploy_modulereleasemeta.sh template-operator regular:$MT_VERSION
```

### Create a Kyma CR and a Remote Secret
### 7. [OPTIONAL] Add the template-operator Module to the Kyma CR and Verify If It Becomes Ready

1. Switch the context for using the KCP cluster:
1. Add the template-operator module to the Kyma CR spec.

```shell
kubectl config use-context k3d-kcp-local
```sh
kubectl config use-context k3d-skr
kubectl get kyma/default -n kyma-system -o yaml | yq e '.spec.modules[0]={"name": "template-operator"}' | kubectl apply -f -
```

2. Generate and apply a sample Kyma CR and its corresponding Secret on KCP:

```shell
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: kyma-sample
namespace: kcp-system
labels:
"operator.kyma-project.io/kyma-name": "kyma-sample"
"operator.kyma-project.io/managed-by": "lifecycle-manager"
data:
config: $(k3d kubeconfig get skr-local | sed 's/0\.0\.0\.0/host.k3d.internal/' | base64 | tr -d '\n')
---
apiVersion: operator.kyma-project.io/v1beta2
kind: Kyma
metadata:
annotations:
skr-domain: "example.domain.com"
name: kyma-sample
namespace: kcp-system
spec:
channel: regular
EOF
1. Verify if the module becomes `Ready`. It takes roughly 1–2 minutes.

```sh
kubectl config use-context k3d-skr
kubectl get kyma/default -n kyma-system -o wide
```

<details>
<summary>Running Lifecycle Manager on a local machine and not on a cluster</summary>
If you are running Lifecycle Manager on your local machine and not as a deployment on a cluster, use the following to create a Kyma CR and Secret:
```shell
cat << EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
name: kyma-sample
namespace: kcp-system
labels:
"operator.kyma-project.io/kyma-name": "kyma-sample"
"operator.kyma-project.io/managed-by": "lifecycle-manager"
data:
config: $(k3d kubeconfig get skr-local | base64 | tr -d '\n')
---
apiVersion: operator.kyma-project.io/v1beta2
kind: Kyma
metadata:
annotations:
skr-domain: "example.domain.com"
name: kyma-sample
namespace: kcp-system
spec:
channel: regular
modules:
- name: template-operator
EOF
```
</details>
### Watcher and Module Installation Verification
Check the Kyma CR events to verify if the `SKRWebhookIsReady` condition is set to `True`.
Also make sure if the state of the `template-operator` is `Ready` and check the overall `state`.
```yaml
status:
activeChannel: regular
conditions:
- lastTransitionTime: "2023-02-28T06:42:00Z"
message: skrwebhook is synchronized
observedGeneration: 1
reason: SKRWebhookIsReady
status: "True"
type: Ready
lastOperation:
lastUpdateTime: "2023-02-28T06:42:00Z"
operation: kyma is ready
modules:
- channel: regular
fqdn: kyma-project.io/module/template-operator
manifest:
apiVersion: operator.kyma-project.io/v1beta2
kind: Manifest
metadata:
generation: 1
name: kyma-sample-template-operator-3685142144
namespace: kcp-system
name: template-operator
state: Ready
template:
apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleTemplate
metadata:
generation: 1
name: moduletemplate-template-operator
namespace: kcp-system
version: 1.2.3
state: Ready
```
1. Remove the module from the Kyma CR spec.

### (Optional) Check the Functionality of the Watcher Component
```sh
kubectl config use-context k3d-skr
kubectl get kyma/default -n kyma-system -o yaml | yq e 'del(.spec.modules[0])' | kubectl apply -f -
```

1. Switch the context to use the SKR cluster:
### 8. [OPTIONAL] Verify Conditions

```shell
kubectl config use-context k3d-skr-local
```
Check the conditions of the Kyma CR in the KCP cluster.

2. Change the channel of the `template-operator` module to trigger a watcher event to KCP:
- `SKRWebhook` to determine if the webhook has been installed to the SKR
- `ModuleCatalog` to determine if the ModuleTemplate CRs and ModuleReleaseMeta CRs haven been synced to the SKR cluster
- `Modules` to determine if the added modules are `Ready`

```yaml
modules:
- name: template-operator
channel: fast
```
```sh
kubectl config use-context k3d-kcp
kubectl get kyma/kyma-sample -n kcp-system -o yaml | yq e '.status.conditions'
```

### Verify logs
### 9. [OPTIONAL] Verify If Watcher Events Reach KCP

1. By watching the `skr-webhook` deployment logs, verify if the KCP request is sent successfully:
1. Flick the channel to trigger an event.

```log
1.6711877286771238e+09 INFO skr-webhook Kyma UPDATE validated from webhook
1.6711879279507768e+09 INFO skr-webhook incoming admission review for: operator.kyma-project.io/v1alpha1, Kind=Kyma
1.671187927950956e+09 INFO skr-webhook KCP {"url": "https://host.k3d.internal:9443/v1/lifecycle-manager/event"}
1.6711879280545895e+09 INFO skr-webhook sent request to KCP successfully for resource default/kyma-sample
1.6711879280546305e+09 INFO skr-webhook kcp request succeeded
```sh
kubectl config use-context k3d-skr
kubectl get kyma/default -n kyma-system -o yaml | yq e '.spec.channel="regular"' | kubectl apply -f -
kubectl get kyma/default -n kyma-system -o yaml | yq e '.spec.channel="fast"' | kubectl apply -f -
```

2. In Lifecycle Manager's logs, verify if the listener is logging messages indicating the reception of a message from the watcher:
1. Verify if Lifecycle Manger received the event in KCP.

```log
{"level":"INFO","date":"2023-01-05T09:21:51.01093031Z","caller":"event/skr_events_listener.go:111","msg":"dispatched event object into channel","context":{"Module":"Listener","resource-name":"kyma-sample"}}
{"level":"INFO","date":"2023-01-05T09:21:51.010985Z","logger":"listener","caller":"controllers/setup.go:100","msg":"event coming from SKR, adding default/kyma-sample to queue","context":{}}
{"level":"INFO","date":"2023-01-05T09:21:51.011080512Z","caller":"controllers/kyma_controller.go:87","msg":"reconciling modules","context":{"controller":"kyma","controllerGroup":"operator.kyma-project.io","controllerKind":"Kyma","kyma":{"name":"kyma-sample","namespace":"default"},"namespace":"default","name":"kyma-sample","reconcileID":"f9b42382-dc68-41d2-96de-02b24e3ac2d6"}}
{"level":"INFO","date":"2023-01-05T09:21:51.043800866Z","caller":"controllers/kyma_controller.go:206","msg":"syncing state","context":{"controller":"kyma","controllerGroup":"operator.kyma-project.io","controllerKind":"Kyma","kyma":{"name":"kyma-sample","namespace":"default"},"namespace":"default","name":"kyma-sample","reconcileID":"f9b42382-dc68-41d2-96de-02b24e3ac2d6","state":"Processing"}}
```sh
kubectl config use-context k3d-kcp
kubectl logs deploy/klm-controller-manager -n kcp-system | grep "event received from SKR"
```

### Cleanup
### 10. [OPTIONAL] Delete the Local Test Clusters

Run the following command to remove the local testing clusters:
Remove the local SKR and KCP test clusters.

```shell
k3d cluster rm kcp-local skr-local
k3d cluster rm kcp skr
```
Loading

0 comments on commit 1fd218e

Please sign in to comment.