Move away from Jobs to Pods #266

lpiwowar · 2024-12-12T17:04:26Z

The test-operator is using Jobs to spawn test pods even though it does
not use any features of this k8s object. Plus usage of the Jobs requires
creation of ServiceAccount in the target namespaces. In order to be able
to create a new, SA the test-oprator has to have a rights to create new
roles and rolebindings which in our case makes the attack surface
larger.

This patch drops the usage of Jobs and moves to Pods.

Depends-On: openstack-k8s-operators/ci-framework#2604

openshift-ci · 2024-12-12T17:04:32Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-12-12T17:04:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lpiwowar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lpiwowar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2024-12-12T21:23:30Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9f2d32ae798a498fb41c1b99fcb8cb1e

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 22m 41s
❌ podified-multinode-edpm-deployment-crc-test-operator TIMED_OUT in 3h 12m 53s

With this PR [1] the test operator dropped the usage of OCP Jobs for spawning of the test pods. This change updates the test-operator role so that it works with the new version of the test-operator that spawns test pods directly through the OCP Pods object. [1] openstack-k8s-operators/test-operator#266

softwarefactory-project-zuul · 2024-12-13T14:31:18Z

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#2604 is needed.

lpiwowar · 2024-12-13T14:31:39Z

recheck

In this PR [1] we dropped the usage of the Jobs by the test-operator. This allows us to drop the rights for: - ServiceAccount managing & creation - Role managing & creation - RoleBinding managing & creation These rights were only needed because Jobs were required to be spawned with an extra ServiceAccount that would have elevated privileges in case the test pods need to run with privileged SecurityContext. [1] openstack-k8s-operators#266 Depends-On: openstack-k8s-operators#266

softwarefactory-project-zuul · 2024-12-13T15:48:23Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8d8a9fbde32c44ddbc4991307290d9e9

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 15m 41s
❌ podified-multinode-edpm-deployment-crc-test-operator FAILURE in 1h 06m 22s

With this PR [1] the test operator dropped the usage of OCP Jobs for spawning of the test pods. This change updates the test-operator role so that it works with the new version of the test-operator that spawns test pods directly through the OCP Pods object. [1] openstack-k8s-operators/test-operator#266

lpiwowar · 2024-12-17T15:01:58Z

recheck

lpiwowar · 2024-12-17T15:03:07Z

/hold

These PRs need to be merged at the same time. The idea is to first get /approved and /lgtm for both of them and then remove /hold at the same time:

lpiwowar · 2024-12-18T12:32:31Z

/test all

lpiwowar · 2024-12-20T09:51:36Z

/test all

With this PR [1] the test operator dropped the usage of OCP Jobs for spawning of the test pods. This change updates the test-operator role so that it works with the new version of the test-operator that spawns test pods directly through the OCP Pods object. [1] openstack-k8s-operators/test-operator#266

softwarefactory-project-zuul · 2025-01-17T12:29:15Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7e844aee32a34fc49f131f3cb5ebff15

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 59m 49s
❌ podified-multinode-edpm-deployment-crc-test-operator FAILURE in 2h 50m 37s

controllers/ansibletest_controller.go

kstrenkova

I went through the code a bit and there are many instances, where the job words/variables are not changed into pod. I will check the functionality next, but I wanted to call attention to this first.

pkg/tobiko/pod.go

controllers/common.go

controllers/horizontest_controller.go

pkg/ansibletest/pod.go

pkg/horizontest/pod.go

pkg/tempest/pod.go

controllers/tobiko_controller.go

controllers/common.go

lpiwowar · 2025-01-27T10:18:13Z

Thank you @kstrenkova! Lot of renaming that needs to be done. I'll take a look. Can you also please review it from the user / functionality perspective?

lpiwowar · 2025-01-27T10:20:07Z

@kstrenkova: Also, please try to use the Review feature on GitHub. You can add the comments during review and then send them all and once with "Request Change".

kstrenkova · 2025-01-27T12:26:55Z

@kstrenkova: Also, please try to use the Review feature on GitHub. You can add the comments during review and then send them all and once with "Request Change".

I actually tried to do that, but once I did Start a Review, it deleted my comments... So I added them later 😅 Maybe if I chose request changes, it would work, I will try it next time :D

lpiwowar

The names should be updated.

controllers/ansibletest_controller.go

controllers/common.go

controllers/tobiko_controller.go

pkg/ansibletest/pod.go

pkg/horizontest/pod.go

pkg/tempest/pod.go

pkg/tobiko/pod.go

softwarefactory-project-zuul · 2025-01-28T11:06:12Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3dcf41acd2954779b64cc74819f4f52f

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 23m 47s
❌ podified-multinode-edpm-deployment-crc-test-operator FAILURE in 1h 12m 04s

In this PR [1] we dropped the usage of the Jobs by the test-operator. This allows us to drop the rights for: - ServiceAccount managing & creation - Role managing & creation - RoleBinding managing & creation These rights were only needed because Jobs were required to be spawned with an extra ServiceAccount that would have elevated privileges in case the test pods need to run with privileged SecurityContext. [1] openstack-k8s-operators#266 Depends-On: openstack-k8s-operators#266

kstrenkova · 2025-01-28T12:29:36Z

As of now, I have tested functionality of this patch as well. It seems to be working with all the CRs and the ownerRefrences are no longer kind: Job. Instead, it's kind: Tempest and other types, respectively. I just want to ask out of curiosity, why it is not kind: Pod? Other than that, it works just fine 😄

softwarefactory-project-zuul · 2025-01-28T12:58:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/81fa1a540f85479889489c26e011e748

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 18m 37s
❌ podified-multinode-edpm-deployment-crc-test-operator FAILURE in 1h 09m 11s

The test-operator is using Jobs to spawn test pods even though it does not use any features of this k8s object. Plus usage of the Jobs requires creation of ServiceAccount in the target namespaces. In order to be able to create a new, SA the test-oprator has to have a rights to create new roles and rolebindings which in our case makes the attack surface larger. This patch drops the usage of Jobs and moves to Pods. Depends-On: openstack-k8s-operators/ci-framework#2604

lpiwowar · 2025-01-29T09:01:33Z

It looks like the CI was not picking up the change. I've tried to merge the two commits together. Let's wait and see.

softwarefactory-project-zuul · 2025-01-29T10:53:19Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3614a08c4caa4c9f80f7c451a63a287b

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 29m 57s
❌ podified-multinode-edpm-deployment-crc-test-operator FAILURE in 1h 12m 28s

openshift-ci bot added the do-not-merge/work-in-progress label Dec 12, 2024

openshift-ci bot added the approved label Dec 12, 2024

lpiwowar mentioned this pull request Dec 13, 2024

[test-operator] Move away from Jobs to Pods openstack-k8s-operators/ci-framework#2604

Closed

lpiwowar force-pushed the remove-jobs-usage branch from cdb82e1 to 9efbbb1 Compare December 13, 2024 14:28

lpiwowar changed the title ~~Move test-operator away from Jobs~~ Move away from Jobs to Pods Dec 13, 2024

lpiwowar mentioned this pull request Dec 13, 2024

Remove unnecessary rights #267

Draft

openshift-ci bot added the do-not-merge/hold label Dec 17, 2024

lpiwowar force-pushed the remove-jobs-usage branch from 9efbbb1 to cd18a0b Compare December 18, 2024 13:26

openshift-merge-robot added the needs-rebase label Jan 15, 2025

lpiwowar force-pushed the remove-jobs-usage branch from cd18a0b to f0e6783 Compare January 17, 2025 09:12

openshift-merge-robot removed the needs-rebase label Jan 17, 2025

This was referenced Jan 17, 2025

[test-operator] Move away from Jobs to Pods openstack-k8s-operators/ci-framework#2663

Closed

[test-operator] Move away from Jobs to Pods openstack-k8s-operators/ci-framework#2664

Draft