KCP Automation - Programming Languages/ Frameworks to use

[kasturinarra]

We plan to automate basic sanity / smoke testing for KCP and researching about programming languages/ frameworks to use. Current suggestions are not to use gingko, bash and use built-in go test or may be python ? Raising the discussion here to see if there are any other better suggestions/ideas about the programming language and frameworks which we can use, Input would be a kubeconfig file we would just export it and run tests which are developed.

[ncdc]

@stevekuznetsov I'm sure you have some opinions here 😄

[ncdc]

Also: these tests are testing as we'd expect end users to interact with the system; namely, with kubectl and our plugins.

[xltian]

@ncdc Hey, Andy, most other OpenShift components e2e testing are being automated with golang and ginkgo, any special reason that ginkgo framework won't be applicable for KCP?

[ncdc]

Hi @xltian, I don't know that I can speak for everyone, but at least some of us have found working with ginkgo to take more effort and to not to be as productive as the regular go testing package. If you can introduce patterns that make it easy to introduce new test and that don't make the ongoing maintenance burden too high, we potentially could consider ginkgo.

@kcp-dev/kcp-prototypers please share your thoughts if you have any. Thanks!

[marun]

I think use of ginkgo can be problematic from a maintainability perspective if the tests end up following the Behavior-Driven Development (BDD) pattern that ginkgo enables. This style of testing is suited to writing tests from an end-user perspective (e.g. when I take this action, I expect this to happen) but does not seem ideal for validating complex distributed systems like kube, openshift or kcp.

That said, I also don't think that go's built-in test framework is ideally suited to that kind of e2e testing either. It's well-suited to small-scale testing (e.g. unit), but there are a number of gaps that I think may become increasingly problematic for the project:

• The process for a given test is isolated to the test package. That forces sharing state or context across a test run to be done external to the test process (e.g. prior to suite invocation or via the filesystem).
• Similarly, each test package generates its own binary. afaink there's no easy way to compose a single binary that is composed of multiple packages - attempting to run go test -c against multiple packages will fail. This complicates the practice of shipping a known-good test binary that can be used by consumers (e.g. qa/customers/etc).

I think it may be worth considering switching the e2e suite to a tool that doesn't have the shortcomings listed above. Ginkgo is one such tool, and perhaps there are others.

Note that a switch to ginkgo wouldn't require adoption of the BDD-style of testing. The kcp project could decide to adopt the parts of ginkgo that are productive (the suite composition and execution parts that address the shortcomings of go test) while retaining the relative simplicity and clarity of the current test suite.

[csams]

If the process is running some shell commands and interpreting the output, I second python for the job.

[kasturinarra]

Notes from yesterday's meeting: patterns that are used to use like gingko in bunch of the openshift qe tests may be we just say roll with it, i will make sure we get feedback and ideally we can make a decision today or tomorrow so that you are not held up any more. Thanks !!
cc: @ncdc

[ncdc]

@kasturinarra do you have an example of some ginkgo tests that are testing the CLI (oc and/or kubectl)?

[ncdc]

@kasturinarra I don't know that we want to mix ginkgo and non-ginkto tests in the test/e2e packages, do we?

[kasturinarra]

@ncdc i am not sure if that would cause any friction, well if we do not want to we could create a new repo.

[ncdc]

If you're adding to test/e2e, I'd say your additions should be aligned with the patterns we're currently using. If you're looking to add different kinds of tests, we could create a separate directory for you (test/functional or test/smoke or test/cli or something like that), and you could use ginkgo in there. Or we can go with a separate repo. @sttts WDYT?

[kasturinarra]

@ncdc i think it should be okay to create a new repo, thanks !!

[ncdc]

https://github.com/kcp-dev/kcp-tests