MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing #6385

danmanor · 2024-05-30T13:58:18Z

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.
Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.
Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

Network: With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we expose static nodePort for each service and configure kind to forward the traffic from localhost:target-port to cluster:nodePort
Local Registry: With Minikube, we were using local registry to store assisted-service image after it was built, for the pod to pull from later. Configuring local registry with kind using podman is not supported and redundant. Instead, I loaded the build image into kind for local use

List all the issues related to this PR

What environments does this code impact?

Automation (CI, tools, etc)
Cloud
Operator Managed Deployments
None

How was this code tested?

assisted-test-infra environment
dev-scripts environment
Reviewer's test appreciated
Waiting for CI to do a full test run
Manual (Elaborate on how it was tested)
No tests needed

Checklist

Title and description added to both, commit and PR.
Relevant issues have been associated (see CONTRIBUTING guide)
This change does not require a documentation update (docstring, docs, README, etc)
Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?
Is there a bug required (and linked) for this change?
Should this PR be backported?

openshift-ci-robot · 2024-05-30T13:58:21Z

openshift-ci-robot · 2024-05-30T14:06:35Z

openshift-ci-robot · 2024-05-30T14:18:58Z

openshift-ci-robot · 2024-05-30T14:19:11Z

codecov · 2024-05-30T14:27:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.45%. Comparing base (f961a0d) to head (ab4e574).
Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6385      +/-   ##
==========================================
+ Coverage   68.43%   68.45%   +0.01%     
==========================================
  Files         246      247       +1     
  Lines       36149    36230      +81     
==========================================
+ Hits        24740    24800      +60     
- Misses       9221     9232      +11     
- Partials     2188     2198      +10

see 10 files with indirect coverage changes

openshift-ci-robot · 2024-05-30T14:43:50Z

@danmanor: This pull request references MGMT-8356 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.

Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.

Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we forward traffic differently - we use contour (https://projectcontour.io/) Ingress controller and Ingress resources to forward traffic from the hostname:80 to the application's services. We also expose port 80 in kind to get the traffic into envoy. This solves the issue in a good way for all components but postgres, which requires a TCP connection which contour can't provide. Therefore, I exposed port 5432as well and forward this traffic toNodePort 30000 (postgresserviceNodePort`).

List all the issues related to this PR

New Feature

Enhancement

Bug fix

Tests

Documentation

CI/CD

What environments does this code impact?

Automation (CI, tools, etc)

Cloud

Operator Managed Deployments

None

How was this code tested?

assisted-test-infra environment

dev-scripts environment

Reviewer's test appreciated

Waiting for CI to do a full test run

Manual (Elaborate on how it was tested)

No tests needed

Checklist

Title and description added to both, commit and PR.

Relevant issues have been associated (see CONTRIBUTING guide)

This change does not require a documentation update (docstring, docs, README, etc)

Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?

Is there a bug required (and linked) for this change?

Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-05-30T14:44:22Z

@danmanor: This pull request references MGMT-8356 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.

Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.

Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we forward traffic differently - we use contour (https://projectcontour.io/) Ingress controller and Ingress resources to forward traffic from the hostname:80 to the application's services. We also expose port 80 in kind to get the traffic into envoy. This solves the issue in a good way for all components but postgres, which requires a TCP connection which contour can't provide. Therefore, I exposed port 5432 as well and forward this traffic to NodePort 30000 (postgres service NodePort).

List all the issues related to this PR

New Feature

Enhancement

Bug fix

Tests

Documentation

CI/CD

What environments does this code impact?

Automation (CI, tools, etc)

Cloud

Operator Managed Deployments

None

How was this code tested?

assisted-test-infra environment

dev-scripts environment

Reviewer's test appreciated

Waiting for CI to do a full test run

Manual (Elaborate on how it was tested)

No tests needed

Checklist

Title and description added to both, commit and PR.

Relevant issues have been associated (see CONTRIBUTING guide)

This change does not require a documentation update (docstring, docs, README, etc)

Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?

Is there a bug required (and linked) for this change?

Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

danmanor · 2024-05-30T14:44:43Z

/cc @rccrdpccl

openshift-ci-robot · 2024-05-30T14:47:04Z

@danmanor: This pull request references MGMT-8356 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.

Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.

Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we forward traffic differently - we use contour (https://projectcontour.io/) Ingress controller and Ingress resources to forward traffic from the hostname:80 to the application's services. We also expose port 80 in kind to get the traffic into envoy. This solves the issue in a good way for all components but postgres, which requires a TCP connection which contour can't provide. Therefore, I exposed port 5432 as well and forward this traffic to NodePort 30000 (postgres service NodePort).

To Do:

metrics tests are still failing locally with kind

List all the issues related to this PR

New Feature

Enhancement

Bug fix

Tests

Documentation

CI/CD

What environments does this code impact?

Automation (CI, tools, etc)

Cloud

Operator Managed Deployments

None

How was this code tested?

assisted-test-infra environment

dev-scripts environment

Reviewer's test appreciated

Waiting for CI to do a full test run

Manual (Elaborate on how it was tested)

No tests needed

Checklist

Title and description added to both, commit and PR.

Relevant issues have been associated (see CONTRIBUTING guide)

This change does not require a documentation update (docstring, docs, README, etc)

Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?

Is there a bug required (and linked) for this change?

Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-05-30T14:50:16Z

@danmanor: This pull request references MGMT-8356 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.

Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.

Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

Network: With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we forward traffic differently - we use contour (https://projectcontour.io/) Ingress controller and Ingress resources to forward traffic from the hostname:80 to the application's services. We also expose port 80 in kind to get the traffic into envoy. This solves the issue in a good way for all components but postgres, which requires a TCP connection which contour can't provide. Therefore, I exposed port 5432 as well and forward this traffic to NodePort 30000 (postgres service NodePort).

Local Registry: With Minikube, we were using local registry to store assisted-service image after it was built, for the pod to pull from later. Configuring local registry with kind using podman is not supported and redundant. Instead, I loaded the build image into kind for local use

To Do:

metrics tests are still failing locally with kind

List all the issues related to this PR

New Feature

Enhancement

Bug fix

Tests

Documentation

CI/CD

What environments does this code impact?

Automation (CI, tools, etc)

Cloud

Operator Managed Deployments

None

How was this code tested?

assisted-test-infra environment

dev-scripts environment

Reviewer's test appreciated

Waiting for CI to do a full test run

Manual (Elaborate on how it was tested)

No tests needed

Checklist

Title and description added to both, commit and PR.

Relevant issues have been associated (see CONTRIBUTING guide)

This change does not require a documentation update (docstring, docs, README, etc)

Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?

Is there a bug required (and linked) for this change?

Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-05-30T14:52:19Z

@danmanor: This pull request references MGMT-8356 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

Overview

This PR is is about using kind cluster using podman provider to deploy assisted-service and its components for local subsystem tests.

Goals

Using kind instead of Minikube for local k8s development can offer several advantages:

Speed and Simplicity: kind starts faster than Minikube because it runs k8s cluster components as podman containers. This can lead to quicker setup and teardown times.

Less Resource-Intensive: Since kind runs within podman, it tends to use less system resources compared to Minikube, which creates a new VM for the k8s cluster. This is a big advantage for running subsystem tests on a PC.

Avoid using KVM: KVM was causing several issues in our deployments.

Major Changes

Network: With Minikube, components we automatically exposed outside the cluster when their LoadBalancer Service resource was applied (by getting External IP). we utilized it to get traffic inside the cluster into the components. this is not the case with kind, so we forward traffic differently - we use contour (https://projectcontour.io/) Ingress controller and Ingress resources to forward traffic from the hostname:80 to the application's services. We also expose port 80 in kind to get the traffic into envoy. This solves the issue in a good way for all components but postgres, which requires a TCP connection which contour can't provide. Therefore, I exposed port 5432 as well and forward this traffic to NodePort 30000 (postgres service NodePort).

Local Registry: With Minikube, we were using local registry to store assisted-service image after it was built, for the pod to pull from later. Configuring local registry with kind using podman is not supported and redundant. Instead, I loaded the build image into kind for local use

To Do:

metrics tests are still failing locally with kind

CI deployment is still damaged

List all the issues related to this PR

New Feature

Enhancement

Bug fix

Tests

Documentation

CI/CD

What environments does this code impact?

Automation (CI, tools, etc)

Cloud

Operator Managed Deployments

None

How was this code tested?

assisted-test-infra environment

dev-scripts environment

Reviewer's test appreciated

Waiting for CI to do a full test run

Manual (Elaborate on how it was tested)

No tests needed

Checklist

Title and description added to both, commit and PR.

Relevant issues have been associated (see CONTRIBUTING guide)

This change does not require a documentation update (docstring, docs, README, etc)

Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

Are the title and description (in both PR and commit) meaningful and clear?

Is there a bug required (and linked) for this change?

Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

danmanor · 2024-05-31T11:07:58Z

/test edge-subsystem-kubeapi-aws

danmanor · 2024-05-31T11:08:03Z

/test edge-subsystem-aws

tools/utils.py

tools/deploy_wiremock.py

danmanor · 2024-06-18T07:38:57Z

/unhold

Makefile

docs/dev/running-test.md

Makefile

danmanor · 2024-06-18T10:33:01Z

/retest

rccrdpccl · 2024-06-18T12:27:43Z

/lgtm
/approve

openshift-ci · 2024-06-18T12:34:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danmanor, rccrdpccl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [danmanor,rccrdpccl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-06-18T17:28:11Z

@danmanor: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-bot · 2024-06-19T01:26:05Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-agent-installer-api-server-container-v4.17.0-202406190043.p0.g0976175.assembly.stream.el9 for distgit ose-agent-installer-api-server.
All builds following this will include this PR.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 30, 2024

openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 30, 2024

openshift-ci bot requested review from jhernand and omertuc May 30, 2024 14:00

danmanor changed the title ~~WIP: NO-ISSUE: Add kind subsystem~~ WIP: NO-ISSUE: Use kind for subsystem tests instead of minikube May 30, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 30, 2024

danmanor changed the title ~~WIP: NO-ISSUE: Use kind for subsystem tests instead of minikube~~ WIP: NO-ISSUE: Use kind for local subsystem tests instead of minikube May 30, 2024

danmanor changed the title ~~WIP: NO-ISSUE: Use kind for local subsystem tests instead of minikube~~ MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing May 30, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 30, 2024

danmanor changed the title ~~MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing~~ WIP: MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing May 30, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 30, 2024

danmanor force-pushed the kind-subsystem branch from 0bb9169 to 6efed48 Compare May 30, 2024 14:07

openshift-ci bot requested a review from rccrdpccl May 30, 2024 14:44

danmanor marked this pull request as draft May 30, 2024 14:52

danmanor force-pushed the kind-subsystem branch from 6efed48 to f1ff665 Compare May 31, 2024 11:07

danmanor force-pushed the kind-subsystem branch from f1ff665 to ff2e342 Compare May 31, 2024 13:13

openshift-ci bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 17, 2024

danmanor force-pushed the kind-subsystem branch from cffb8a4 to ae6ffd0 Compare June 17, 2024 14:31

rccrdpccl reviewed Jun 17, 2024

View reviewed changes

tools/utils.py Outdated Show resolved Hide resolved

rccrdpccl reviewed Jun 17, 2024

View reviewed changes

tools/utils.py Outdated Show resolved Hide resolved

rccrdpccl reviewed Jun 17, 2024

View reviewed changes

tools/utils.py Show resolved Hide resolved

rccrdpccl reviewed Jun 17, 2024

View reviewed changes

tools/deploy_wiremock.py Show resolved Hide resolved

danmanor force-pushed the kind-subsystem branch 2 times, most recently from 961f1cc to fe97e3b Compare June 17, 2024 15:09

danmanor requested a review from rccrdpccl June 17, 2024 15:09

danmanor force-pushed the kind-subsystem branch from fe97e3b to 8e30455 Compare June 18, 2024 07:38

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 18, 2024

rccrdpccl reviewed Jun 18, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

rccrdpccl reviewed Jun 18, 2024

View reviewed changes

docs/dev/running-test.md Show resolved Hide resolved

danmanor requested a review from rccrdpccl June 18, 2024 09:29

danmanor force-pushed the kind-subsystem branch from 8e30455 to c345228 Compare June 18, 2024 09:53

rccrdpccl reviewed Jun 18, 2024

View reviewed changes

Makefile Show resolved Hide resolved

rccrdpccl reviewed Jun 18, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

danmanor requested a review from rccrdpccl June 18, 2024 11:47

MGMT-8356: Deploy assisted-service and its components to kind cluster

ab4e574

danmanor force-pushed the kind-subsystem branch from c345228 to ab4e574 Compare June 18, 2024 12:15

openshift-ci bot assigned rccrdpccl Jun 18, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 18, 2024

openshift-merge-bot bot merged commit 0976175 into openshift:master Jun 18, 2024
14 checks passed

danmanor mentioned this pull request Jul 23, 2024

MGMT-18451: Enable debugging assisted-service on kind #6613

Merged

20 tasks

MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing #6385

MGMT-8356: Deploy assisted-service and its components to kind cluster instead of minikube for local subsystem testing #6385

Conversation

danmanor commented May 30, 2024 • edited Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Chages

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

codecov bot commented May 30, 2024 • edited Loading

Codecov Report

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

danmanor commented May 30, 2024

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

Reviewers Checklist

openshift-ci-robot commented May 30, 2024 • edited by openshift-ci bot Loading

Overview

Goals

Major Changes

List all the issues related to this PR

What environments does this code impact?

How was this code tested?

Checklist

danmanor commented May 30, 2024 •

edited

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

codecov bot commented May 30, 2024 •

edited

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 30, 2024 •

edited by openshift-ci bot

Loading