-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Testing for Plugins #5724
Comments
Testing strategySo with Phase 2 Plugins on the way, it seems like there is the potential to have 2 kinds of plugins:
Traditional pluginsWith the traditional plugins (ones that have to be pulled in and built by operator-sdk like Helm/Ansible/Java), the e2e testing strategy is much more difficult because it is a library that is being imported by operator-sdk and then run. I believe a potential solution for this could be to create a plugin testing package that these traditional plugins can use to test with an in memory filesystem. This would allow for these traditional plugins to be able to be e2e tested programmatically and the in-memory file system can be checked to make sure everything was scaffolded as expected. I think this method would require some upstream changes in Kubebuilder to allow the modification of the filesystem used as part of the CLI. Phase 2 PluginsWith the Phase 2 Plugins, it will be much easier to e2e test these types of plugins because they will be binaries and with the architecture of the Phase 2 Plugins they will simply read JSON in via stdin and output JSON via stdout. Because these will be much easier to test, is this something that we would want to create some form of tooling to make the testing a bit easier or leave this up to the developers of Phase 2 Plugins to handle themselves? One potential way we could help is by creating a few libraries for common programming languages that essentially make it easier to create inputs and parse outputs for the Phase 2 Plugins by mapping the Phase 2 Plugin input/output schemas to objects. |
Hi @everettraven, It exists already. You can check the lib: https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/utils Then, you can check some examples:
Note that any external plugins must have a way to be executed by themselves, which means using the Kubebuilder lib and doing the CLI implementation. Therefore, the external plugin should have its own binary and can indeed have a testdata dir with samples such as we build in both projects. However, (IMHO) would be nice to document it and add examples here: https://book.kubebuilder.io/plugins/plugins.html. We could create there a new section "Writing tests for your own plugins" then we also could have 2 subsections: generating testdata samples and writing e2e tests. WDYT? But I think that would be also a task for Kubebuilder not SDK. |
@camilamacedo86 interesting.... I raised this issue because of a conversation had in the OSDK community meeting and a separate offline conversation. @jmrodri was a part of that offline conversation and brought up that testing scaffolds was a problem for the java-operator-plugins repo and the helm-operator-plugins repo. Unfortunately there was a lot of other information in that conversation so I have some limited notes regarding the actual testing portion of the conversation. @jmrodri Would you mind elaborating a bit more on the testing issue? |
After digging around a bit I think I am understanding a bit more what the actual point of this issue should be. It seems like the issue is that it is difficult to write lightweight tests to see exactly what files and their contents are being scaffolded when the plugin is run via something like @jmrodri does this sound correct? |
Hi @everettraven,
So, if you are looking for creating samples under the test data directory. Then, see that we use the KB test utils and other common utils provided by KB to generate the samples on SDK: https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples/internal. Note that we call the tool binary and afterwords we use the utils provided to insert/replace content on the scaffold. In both projects, the samples are updated with the code changes made and checked via the CI to ensure that they will always reflect the latest state/version of what is implemented in the master. We update then when we call the target Regarding the goal of this issue, wdyt about add here its acceptance criteria here? So that we can easier understand what is expected to get done. Ps.: I raised an issue for Kubebuilder (kubernetes-sigs/kubebuilder#2677 ) for we doc how to write tests for the plugins. It was a good point identified for you (IHMO). |
Hi @camilamacedo86 ,
Thanks for all this information! At the moment, I'm not sure what the acceptance criteria would be for this. I think it may be best to address this again during the SDK community meeting on May 9th and get more thoughts on this. @jmrodri had mentioned something about e2e testing for plugins to me and I think more thoughts on this from him would help with understanding what the problem is. I should have spoke with him more regarding it before raising this issue to understand the problem a little bit better, but hopefully during the next community meeting we can get a clear direction for this issue. |
More information was provided during the community meeting. The goal of this issue is to work on getting the e2e tests for the plugins that are in external repositories to be able to run the e2e tests themselves instead of the e2e tests for them residing within the operator-sdk repository. @camilamacedo86 provided some good resources to look into for determining the best way to do this. This issue will continue to be used for discussion of different points and ideas until we come to a solution that defines the path that is going to be taken to get this accomplished. |
So I spent some time looking around at the current ways that testdata is generated and how e2e tests are written. The Kubebuilder library for writing the e2e tests and generating test data are nice, but I feel there is some room for some improvements that I think would make it easier to implement testdata generation and e2e tests in the future. I have created a PoC to test and validate some of these ideas. It can be found at: Pain pointsTest Data GenerationCurrently to create test data new struct is created that implements a few methods that are used to generate a sample project and then make modifications to the files. In the Operator SDK test data generation this process of creating a struct for generating a sample is repeated for each plugin supported and tested by the Operator SDK's e2e tests. This can be seen in the following files:
They all share a very similar structure and way of creating a sample project for testing purposes, albeit having slightly different internal implementations. To address this I propose that we create some kind of interface that allows for an easier way to generate sample projects without having to write a new struct and generation method for every sample project. An interface for these sample projects could be something like: type Sample interface {
CommandContext() command.CommandContext
Name() string
GenerateInit() error
GenerateApi() error
GenerateWebhook() error
} This Then there can be a generator tool that takes a list of these type Generator interface {
GenerateSamples(samples ...samples.Sample) error
} A PoC for a generic Generator that allows for some flexibility in the running of the generation logic can be found here: A couple examples showing how this could simplify the process of test data generation: A single sample generated: simpleSample := samples.NewGenericSample(
samples.WithBinary("/usr/local/bin/operator-sdk"),
samples.WithDomain("sample.com"),
samples.WithGvk(schema.GroupVersionKind{
Group: "simple",
Version: "v1alpha1",
Kind: "Sample",
}),
samples.WithName("simple-sample"),
samples.WithExtraApiOptions("--resource", "--controller"),
)
generator := generator.NewGenericGenerator(
generator.WithNoWebhook(),
)
err := generator.GenerateSamples(simpleSample) Full runable example go file for above code can be found at: https://github.com/everettraven/plugin-testing-poc/blob/main/examples/simple-generate/main.go Multiple samples generated: simpleGoSample := samples.NewGenericSample(
samples.WithBinary("/usr/local/bin/operator-sdk"),
samples.WithDomain("simple.go.com"),
samples.WithGvk(schema.GroupVersionKind{
Group: "simplego",
Version: "v1alpha1",
Kind: "GoSample",
}),
samples.WithName("go-simple-sample"),
samples.WithExtraApiOptions("--resource", "--controller"),
)
simpleHelmSample := samples.NewGenericSample(
samples.WithBinary("/usr/local/bin/operator-sdk"),
samples.WithDomain("simple.helm.com"),
samples.WithGvk(schema.GroupVersionKind{
Group: "simplehelm",
Version: "v1alpha1",
Kind: "HelmSample",
}),
samples.WithName("helm-simple-sample"),
samples.WithPlugins("helm"),
)
simpleAnsibleSample := samples.NewGenericSample(
samples.WithBinary("/usr/local/bin/operator-sdk"),
samples.WithDomain("simple.ansible.com"),
samples.WithGvk(schema.GroupVersionKind{
Group: "simpleansible",
Version: "v1alpha1",
Kind: "AnsibleSample",
}),
samples.WithName("ansible-simple-sample"),
samples.WithPlugins("ansible"),
)
generator := generator.NewGenericGenerator(
generator.WithNoWebhook(),
)
err := generator.GenerateSamples(simpleGoSample, simpleHelmSample, simpleAnsibleSample) Full runable example go file for above code can be found at: https://github.com/everettraven/plugin-testing-poc/blob/main/examples/multiple-samples/main.go E2E TestingThere seems to be a lot of reliance of the Kubebuilder E2E I feel like this limits the ability to extend e2e tests to relying on using and creating a new separate implementation of the One of the common fields that is used from the Another common field that is used is the type CommandContext interface {
Env() []string
Dir() string
Stdin() io.Reader
Run(cmd *exec.Cmd, path ...string) ([]byte, error)
} An example general implementation can be found at: https://github.com/everettraven/plugin-testing-poc/blob/04f59e269bd089f1449f5643d537d099fb094ab7/pkg/command/command.go#L19-L89 The other most common field I noticed was being used was type Kubectl interface {
CommandContext() command.CommandContext
Namespace() string
ServiceAccount() string
// Actual functions
Command(options ...string) (string, error)
CommandInNamespace(options ...string) (string, error)
Apply(inNamespace bool, options ...string) (string, error)
Get(inNamespace bool, options ...string) (string, error)
Delete(inNamespace bool, options ...string) (string, error)
Logs(inNamespace bool, options ...string) (string, error)
Wait(inNamespace bool, options ...string) (string, error)
Version() (KubernetesVersion, error)
} This would allow for multiple different implementations of this to exist, not limiting e2e tests to using The An example of this suggestion can be found at: https://github.com/everettraven/plugin-testing-poc/blob/main/pkg/e2e/helpers.go An example e2e test with these suggested changes was created and mimics the current Operator SDK e2e test for a Go operator. It can be found here: https://github.com/everettraven/plugin-testing-poc/blob/main/examples/e2e/go/e2e_test.go I feel like these changes would make it significantly easier to write e2e tests for plugins that are external from the Operator SDK or Kubebuilder repo while making the e2e tests easier to extend than they are currently. I also believe it would allow Phase 2 Plugins that are written to have e2e tests written using this library (assuming the e2e tests are written with Go). I think if it was deemed necessary multiple versions of this library could be created to help with e2e testing for Phase 2 Plugins written in a language other than Go. @asmacdo @jmrodri @camilamacedo86 I would love to get your thoughts on this! |
Created a draft PR to show a PoC of this testing library/package update here: #5813 |
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
@openshift-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Feature Request
Describe the problem you need a feature to resolve.
There is currently not an easy way to e2e test a plugin without writing all the tests within operator-sdk. This makes it difficult to e2e test a plugin that is in an external repository.
Describe the solution you'd like.
First thought was to create a Go library that could be used to test a plugin, however this may be something that would need to be implemented upstream in Kubebuilder. This approach also will likely not be viable in the future with the introduction of Phase 2 Plugins as plugins will be able to be written in any language.
This issue can be used as a place of discussion as to how we may want to approach this, or with the introduction of Phase 2 Plugins if this is something worth pursuing or leaving it up to the developers of an external plugin to implement their own form of e2e testing.
The text was updated successfully, but these errors were encountered: