chore: migrate from kuttl to chainsaw

[eddycharly]

Migrate from kuttl to chainsaw.

Chainsaw is similar to kuttl but a lot more flexible and actively maintained.

If you need more infos 👇

• GitHub: https://github.com/kyverno/chainsaw
• Docs: https://kyverno.github.io/chainsaw
• Slack: https://kubernetes.slack.com/archives/C067LUFL43U

[eddycharly]

BTW this file is not valid

tempo-operator/tests/e2e-openshift/red-metrics/03-assert.yaml

Lines 52 to 57 in 46c037b

	name: tempo-redmetrics
	apiVersion: kuttl.dev/v1beta1
	kind: TestAssert
	commands:
	- command: /bin/sh -c "kubectl get --namespace $NAMESPACE tempo redmetrics -o jsonpath='{.status.conditions[?(@.type==\"Ready\")].status}' | grep True"

[pavolloffay]

hi @eddycharly thanks for the PR. Is there any statement in kuttl project about not being activelly maintained?

chainsaw seems to be only 3-4 old project (looking at releases) which concerns me. Is there any evidence of adoption by operaotr-sdk/kubebuilder?

[eddycharly]

Hi @pavolloffay

Chainsaw was created because kuttl stopped to be actively maintained (it's actually a man community now).

We are also solving most of the kuttl limitations that exist today and wrote a command to automatically migrate from kuttl to chainsaw.

We presented chainsaw to kubebuilder/operator-sdk folks last week and are discussing how to collaborate:

• chainsaw will be added to the operator-sdk docs soon docs: add note about chainsaw operator-framework/operator-sdk#6675
• we are evaluating the creation of a kubebuilder plugin

I agree this is a relatively young project but it has already been adopted by a couple of e2e tests heavy projects (kyverno, keptn, fairwinds, opentelementry-operator is the next one, and there is a PR for grafana-operator grafana/grafana-operator#1406).

This is a solid kuttl replacement, i can change the PR to run both tools side by side if you prefer but maintaining two identical test suites is a pain.

[andreasgerstmayr]

This is a solid kuttl replacement, i can change the PR to run both tools side by side if you prefer but maintaining two identical test suites is a pain.

Could this be a drop-in replacement? I.e. we can just replace the kuttl binary with chainsaw, and it works as-is?
Once we add chainsaw-only features, we cannot migrate back anymore of course. But it'd make adoption much easier. For example podman did this, you can do alias docker=podman and it just works for most users.

[eddycharly]

Could this be a drop-in replacement?

Unfortunately no, cli args are not exactly the same and chainsaw doesn't support TestStep or TestAssert resources.
That's the reason why we created a migration command.

It would be nice to let the workflows run and look at the result, happy to discuss it more if i have a chance to convince you :)

Here is the recording of the last KB meeting where we presented chainsaw if you want to see it in action https://www.youtube.com/watch?v=Ejof-wtAdQM

[eddycharly]

Grafana-operator adopted chainsaw today grafana/grafana-operator#1406 🎉 ❤️

[eddycharly]

@pavolloffay @andreasgerstmayr did you have a chance to look at this ?

[eddycharly]

@pavolloffay I think we met today during the opentelemetry-operator SIG meeting. I hope you loved chainsaw :)

[pavolloffay]

LGTM I am fine with merging this. @andreasgerstmayr WDYT?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.31%. Comparing base (10f1c2b) to head (dd7cc63).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #786   +/-   ##
=======================================
  Coverage   75.31%   75.31%           
=======================================
  Files          89       89           
  Lines        6369     6369           
=======================================
  Hits         4797     4797           
  Misses       1342     1342           
  Partials      230      230           

Flag	Coverage Δ
unittests	75.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[IshwarKanse]

Does the var $NAMESPACE work here same as KUTTL ? I need to check how the Chainsaw tests work when run from inside a pod as the OpenShift CI uses Prow CI to run the jobs and the tests are run from inside a pod.

[eddycharly]

Yes $NAMESPACE works the same as in KUTTL.
Most scripts can be replaced with asserts/apply operations in chainsaw, i will look deeper at your tests once this PR merges to remove scripts when possible.

[eddycharly]

I'll fix the failing tests soon, thanks for triggering the workflow.

[eddycharly]

@pavolloffay @IshwarKanse i pushed a fix 🤞

[eddycharly]

I'm working on a PR to fix the OpenTelemetry Operator tests, also to get them working with OpenShift CI.

@IshwarKanse please ping me on the PR, there's something i don't understand fully here.

[eddycharly]

We released v0.1.6 today, fixing an important issue related to concurrency, shall i update this PR ?

[IshwarKanse]

@eddycharly yeah sure. Please update the version. Thanks.

[eddycharly]

@IshwarKanse done

[rubenvp8510]

I'm fine with this change.

[eddycharly]

I took the time to remove scripts from upgrade tests, converting

    - script:
        content: kubectl get --namespace $NAMESPACE tempo simplest -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' | grep True

to an assertion tree equivalent

    - assert:
        resource:
          apiVersion: tempo.grafana.com/v1alpha1
          kind: TempoStack
          metadata:
            name: simplest
          status:
            (length(conditions[?(type == 'Ready' && status == 'True')]) == 0): false

[eddycharly]

This would work too, maybe better than using length

    - assert:
        resource:
          apiVersion: tempo.grafana.com/v1alpha1
          kind: TempoStack
          metadata:
            name: simplest
          status:
            "(contains(conditions[*].{type: type, status: status}, {type: 'Ready', status: 'True'}))": true

[IshwarKanse]

@eddycharly Can you resolve this so that we can merge this PR. Auto-merge is enabled.

- NoCluster false
Loading tests...
Error: failed to parse document (spec.steps[8].try[2].get: Invalid value: value provided for unknown field
spec.steps[9].try[1].get: Invalid value: value provided for unknown field
spec.steps[9].try[2].get: Invalid value: value provided for unknown field)
make: *** [Makefile:372: e2e-upgrade] Error 1
Error: Process completed with exit code 2.

Once merged I'll retest on a OpenShift cluster.

[eddycharly]

@IshwarKanse 👀

[eddycharly]

@IshwarKanse can you retry ?

I would be in, but we may want to bring this up on a sync meeting before pressing the button?

I thought it was not going to be merged until a meeting happens 🤷

[eddycharly]

This would work too, maybe better than using length

    - assert:
        resource:
          apiVersion: tempo.grafana.com/v1alpha1
          kind: TempoStack
          metadata:
            name: simplest
          status:
            "(contains(conditions[*].{type: type, status: status}, {type: 'Ready', status: 'True'}))": true

Hey folks, i didn't like the solution above and needed something similar while working on something else, there's a better solution:

    - assert:
        resource:
          apiVersion: tempo.grafana.com/v1alpha1
          kind: TempoStack
          metadata:
            name: simplest
          status:
            (conditions[?type == 'Ready']):
            - status: 'True'

[eddycharly]

If one can trigger the workflow 🙏

What are the next steps ? Any idea when this could potentially land in ? 🤞

[IshwarKanse]

@eddycharly We discussed it internally and decided to merge. cc @frzifus @andreasgerstmayr

[eddycharly]

Thanks folks!
I'm vacationing this week, i will see how to improve tests and open follow up PRs next week.

[eddycharly]

BTW if something breaks or you need help with anything feel free to ping me on slack.