Merge pull request #2934 from Kavinjsir/docs/test-plugin

📖 : Write tests for your own plugin
kubernetes-sigs · Sep 29, 2022 · d7eae60 · d7eae60
2 parents 6b77d9c + e4a0ca9
commit d7eae60
Show file tree

Hide file tree

Showing 5 changed files with 194 additions and 63 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -62,6 +62,63 @@ Following the targets that can be used to test your changes locally.
 
 **NOTE** To use the `make lint` is required to install `golangci-lint` locally. More info: https://github.com/golangci/golangci-lint#install
 
+### Test Plugin
+
+If your intended PR creates a new plugin, make sure the PR also provides test cases. Testing should include:
+
+1. `e2e tests` to validate the behavior of the proposed plugin.
+2. `sample projects` to verify the scaffolded output from the plugin.
+
+#### 1. Plugin E2E Tests
+
+All the plugins provided by Kubebuilder should be validated through `e2e-tests` across multiple platforms.
+
+Current Kubebuilder provides the testing framework that includes testing code based on [ginkgo](https://github.com/onsi/ginkgo), [Github Actions](https://github.com/Kavinjsir/kubebuilder/blob/docs%2Ftest-plugin/.github/workflows/testdata.yml) for unit tests, and multiple env tests driven by [test-infra](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/kubebuilder/kubebuilder-presubmits.yaml).
+
+To fully test the proposed plugin:
+
+1. Create a new package(folder) under `test/e2e/<your-plugin>`.
+2. Create [e2e_suite_test.go](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/v4/e2e_suite_test.go), which imports the necessary testing framework.
+3. Create `generate_test.go` ([ref](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/v4/generate_test.go)). That should:
+   - Introduce/Receive a `TextContext` instance
+   - Trigger the plugin's bound subcommands. See [Init](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L213), [CreateAPI](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.6.0/test/e2e/utils/test_context.go#L222)
+   - Use [PluginUtil](https://pkg.go.dev/sigs.k8s.io/kubebuilder/v3/pkg/plugin/util) to verify the scaffolded outputs. See [InsertCode](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/pkg/plugin/util/util.go#L67), [ReplaceInFile](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.6.0/pkg/plugin/util/util.go#L196), [UncommendCode](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.6.0/pkg/plugin/util/util.go#L86)
+4. Create `plugin_cluster_test.go` ([ref](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/v4/plugin_cluster_test.go)). That should:
+
+   - 4.1. Setup testing environment, e.g:
+
+     - Cleanup environment, create temp dir. See [Prepare](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L97)
+     - If your test will cover the provided features then, ensure that you install prerequisites CRDs: See [InstallCertManager](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L138), [InstallPrometheusManager](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.6.0/test/e2e/utils/test_context.go#L171)
+
+   - 4.2. Run the function from `generate_test.go`.
+
+   - 4.3. Further make sure the scaffolded output works, e.g:
+
+     - Execute commands in your `Makefile`. See [Make](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L240)
+     - Temporary load image of the testing controller. See [LoadImageToKindCluster](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L283)
+     - Call Kubectl to validate running resources. See [utils.Kubectl](https://pkg.go.dev/sigs.k8s.io/kubebuilder/v3/test/e2e/utils#Kubectl)
+
+   - 4.4. Delete temporary resources after testing exited, e.g:
+     - Uninstall prerequisites CRDs: See [UninstallPrometheusOperManager](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L183)
+     - Delete temp dir. See [Destroy](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/utils/test_context.go#L255)
+
+5. Add the command in [test/e2e/plugin](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/e2e/setup.sh#L65) to run your testing code:
+
+```shell
+go test $(dirname "$0")/<your-plugin-test-folder> $flags -timeout 30m
+```
+
+#### 2. Sample Projects from the Plugin
+
+It is also necessary to test consistency of the proposed plugin across different env and the integration with other plugins.
+
+This is performed by generating sample projects based on the plugins. The CI workflow defined in Github Action would validate the availability and the consistency.
+
+See:
+
+- [test/testdata/generated.sh](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/test/testdata/generate.sh#L144)
+- [make generate](https://github.com/kubernetes-sigs/kubebuilder/blob/v3.7.0/Makefile#L70)
+
 ## PR Process
 
 See [VERSIONING.md](VERSIONING.md) for a full description. TL;DR:

diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md
@@ -16,7 +16,7 @@
   - [Adding a new API](./cronjob-tutorial/new-api.md)
   - [Designing an API](./cronjob-tutorial/api-design.md)
 
-      - [A Brief Aside: What's the rest of this stuff?](./cronjob-tutorial/other-api-files.md)
+    - [A Brief Aside: What's the rest of this stuff?](./cronjob-tutorial/other-api-files.md)
 
   - [What's in a controller?](./cronjob-tutorial/controller-overview.md)
   - [Implementing a controller](./cronjob-tutorial/controller-implementation.md)
@@ -39,7 +39,7 @@
   - [Hubs, spokes, and other wheel metaphors](./multiversion-tutorial/conversion-concepts.md)
   - [Implementing conversion](./multiversion-tutorial/conversion.md)
 
-      - [and setting up the webhooks](./multiversion-tutorial/webhooks.md)
+    - [and setting up the webhooks](./multiversion-tutorial/webhooks.md)
 
   - [Deployment and Testing](./multiversion-tutorial/deployment.md)
 
@@ -50,20 +50,22 @@
 
   - [Using a custom type](./component-config-tutorial/custom-type.md)
 
-      - [Adding a new Config Type](./component-config-tutorial/config-type.md)
-      - [Updating main](./component-config-tutorial/updating-main.md)
-      - [Defining your Custom Config](./component-config-tutorial/define-custom-config.md)
+    - [Adding a new Config Type](./component-config-tutorial/config-type.md)
+    - [Updating main](./component-config-tutorial/updating-main.md)
+    - [Defining your Custom Config](./component-config-tutorial/define-custom-config.md)
+
 ---
 
 - [Migrations](./migrations.md)
 
   - [Kubebuilder v1 vs v2](./migration/v1vsv2.md)
 
-      - [Migration Guide](./migration/legacy/migration_guide_v1tov2.md)
+    - [Migration Guide](./migration/legacy/migration_guide_v1tov2.md)
 
   - [Kubebuilder v2 vs v3](./migration/v2vsv3.md)
-      - [Migration Guide](./migration/migration_guide_v2tov3.md)
-      - [Migration by updating the files](./migration/manually_migration_guide_v2_v3.md)
+
+    - [Migration Guide](./migration/migration_guide_v2tov3.md)
+    - [Migration by updating the files](./migration/manually_migration_guide_v2_v3.md)
 
   - [Single Group to Multi-Group](./migration/multi-group.md)
 
@@ -82,12 +84,12 @@
     - [Webhooks for Core Types](reference/webhook-for-core-types.md)
   - [Markers for Config/Code Generation](./reference/markers.md)
 
-      - [CRD Generation](./reference/markers/crd.md)
-      - [CRD Validation](./reference/markers/crd-validation.md)
-      - [CRD Processing](./reference/markers/crd-processing.md)
-      - [Webhook](./reference/markers/webhook.md)
-      - [Object/DeepCopy](./reference/markers/object.md)
-      - [RBAC](./reference/markers/rbac.md)
+    - [CRD Generation](./reference/markers/crd.md)
+    - [CRD Validation](./reference/markers/crd-validation.md)
+    - [CRD Processing](./reference/markers/crd-processing.md)
+    - [Webhook](./reference/markers/webhook.md)
+    - [Object/DeepCopy](./reference/markers/object.md)
+    - [RBAC](./reference/markers/rbac.md)
 
   - [controller-gen CLI](./reference/controller-gen.md)
   - [completion](./reference/completion.md)
@@ -97,7 +99,7 @@
 
   - [Metrics](./reference/metrics.md)
 
-      - [Reference](./reference/metrics-reference.md)
+    - [Reference](./reference/metrics-reference.md)
 
   - [Makefile Helpers](./reference/makefile-helpers.md)
   - [Project config](./reference/project-config.md)
@@ -107,20 +109,21 @@
 - [Plugins][plugins]
 
   - [Available Plugins](./plugins/available-plugins.md)
-      - [go/v2 (Deprecated)](./plugins/go-v2-plugin.md)
-      - [go/v3 (Default init scaffold)](./plugins/go-v3-plugin.md)
-      - [go/v4-alpha](./plugins/go-v4-plugin.md)
-      - [kustomize/v1](./plugins/kustomize-v1.md)
-      - [kustomize/v2-alpha](./plugins/kustomize-v2-alpha.md)
-      - [declarative/v1](./plugins/declarative-v1.md)
-      - [grafana/v1-alpha](./plugins/grafana-v1-alpha.md)
-      - [deploy-image/v1-alpha](./plugins/deploy-image-plugin-v1-alpha.md)
+    - [go/v2 (Deprecated)](./plugins/go-v2-plugin.md)
+    - [go/v3 (Default init scaffold)](./plugins/go-v3-plugin.md)
+    - [go/v4-alpha](./plugins/go-v4-plugin.md)
+    - [kustomize/v1](./plugins/kustomize-v1.md)
+    - [kustomize/v2-alpha](./plugins/kustomize-v2-alpha.md)
+    - [declarative/v1](./plugins/declarative-v1.md)
+    - [grafana/v1-alpha](./plugins/grafana-v1-alpha.md)
+    - [deploy-image/v1-alpha](./plugins/deploy-image-plugin-v1-alpha.md)
   - [Extending the CLI](./plugins/extending-cli.md)
   - [Creating your own plugins](./plugins/creating-plugins.md)
+  - [Testing your own plugins](./plugins/testing-plugins.md)
   - [Plugins Versioning](./plugins/plugins-versioning.md)
 
----  
-[Appendix: The TODO Landing Page](./TODO.md)
+---
 
+[Appendix: The TODO Landing Page](./TODO.md)
 
 [plugins]: ./plugins/plugins.md
diff --git a/docs/book/src/plugins/creating-plugins.md b/docs/book/src/plugins/creating-plugins.md
@@ -25,54 +25,42 @@ If you do not want to develop your plugin using Golang, you can follow its stand
 
 ```sh
 kubebuilder init --plugins=kustomize
-``` 
+```
 
 Then you can, for example, create your implementations for the sub-commands `create api` and `create webhook` using your language of preference.
 
 <aside class="note">
 <h1>Why use the Kubebuilder style?</h1>
 
-Kubebuilder and SDK are both broadly adopted projects which leverage the [controller-runtime][controller-runtime] project.  They both allow users to build solutions using the [Operator Pattern][operator-pattern] and follow common standards.
+Kubebuilder and SDK are both broadly adopted projects which leverage the [controller-runtime][controller-runtime] project. They both allow users to build solutions using the [Operator Pattern][operator-pattern] and follow common standards.
 
 Adopting these standards can bring significant benefits, such as joining forces on maintaining the common standards as the features provided by Kubebuilder and take advantage of the contributions made by the community. This allows you to focus on the specific needs and requirements for your plugin and use-case.
 
 And then, you will also be able to use custom plugins and options currently or in the future which might to be provided by these projects as any other which decides to persuade the same standards.
 
 </aside>
 
-## Custom Plugins 
+## Custom Plugins
 
-Note that users are also able to use plugins to customize their scaffold and address specific needs. See that Kubebuilder provides the [declarative][declarative-code] plugin which can be used when for example an API is scaffold:
+Note that users are also able to use plugins to customize their scaffolds and address specific needs.
 
-```sh
-kubebuider create api [options] --plugins=go/v3,declarative/v1
-``` 
+See that Kubebuilder provides the [deploy-image][deploy-image] plugin that allows the user to create the controller & CRs which will deploy and manage an image on the cluster:
 
-This plugin will perform a custom scaffold using the [kubebuilder declarative pattern][kubebuilder-declarative-pattern].
+```sh
+kubebuilder create api --group example.com --version v1alpha1 --kind Memcached --image=memcached:1.6.15-alpine --image-container-command="memcached,-m=64,modern,-v" --image-container-port="11211" --run-as-user="1001" --plugins="deploy-image/v1-alpha"
+```
 
-In this way, by [Extending the Kubebuilder CLI][extending-cli], you can also create custom plugins such this one. Feel free to check its implementation in [`pkg/plugins/golang/declarative`][declarative-code].
+This plugin will perform a custom scaffold following the [Operator Pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator).
 
-## Future vision for Kubebuilder Plugins 
+Another example is the [grafana][grafana] plugin that scaffolds a new folder container manifests to visualize operator status on Grafana Web UI:
 
-As the next steps for the plugins, its possible to highlight three initiatives so far, which are:
+```sh
+kubebuilder edit --plugins="grafana.kubebuilder.io/v1-alpha"
+```
 
-- [Plugin phase 2.0][plugin-2.0]: allow the Kubebuilder CLI or any other CLI, which is [Extending the Kubebuilder CLI][extending-cli], to discover external plugins, in this way, allow the users to use these external options as helpers to perform the scaffolds with the tool.   
-- [Config-gen][config-gen]: the config-gen option has been provided as an alpha option in the Kubebuilder CLI(`kubebuilder alpha config-gen`) to encourage its contributions. The idea of this option would simplify the config scaffold. For further information see its [README][config-gen-readme].
-- [New Plugin (`deploy-image.go.kubebuilder.io/v1beta1`) to generate code][new-plugin-gen]: its purpose is to provide an arch-type that will scaffold the APIs and Controllers with the required code to deploy and manage solutions on the cluster. 
+In this way, by [Extending the Kubebuilder CLI][extending-cli], you can also create custom plugins such this one.
 
-Please, feel to contribute with them as well. Your contribution to the project is very welcome.  
+Feel free to check the implementation under:
 
-[sdk-cli-pkg]: https://github.com/operator-framework/operator-sdk/blob/master/internal/cmd/operator-sdk/cli/cli.go
-[sdk-ansible]: https://github.com/operator-framework/operator-sdk/tree/master/internal/plugins/ansible/v1
-[sdk-helm]: https://github.com/operator-framework/operator-sdk/tree/master/internal/plugins/helm/v1
-[operator-pattern]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
-[controller-runtime]: https://github.com/kubernetes-sigs/controller-runtime
-[plugin-2.0]: https://github.com/kubernetes-sigs/kubebuilder/issues/1378
-[config-gen-readme]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/cli/alpha/config-gen/README.md
-[config-gen]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/cli/alpha/config-gen
-[plugins-phase1-design-doc-1.5]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1-5.md
-[extending-cli]: extending-cli.md
-[new-plugin-gen]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/code-generate-image-plugin.md
-[kubebuilder-declarative-pattern]: https://github.com/kubernetes-sigs/kubebuilder-declarative-pattern
-[declarative-code]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugins/golang/declarative
-[sdk]: https://github.com/operator-framework/operator-sdk
+- deploy-image: <https://github.com/kubernetes-sigs/kubebuilder/tree/v3.7.0/pkg/plugins/golang/deploy-image/v1alpha1>
+- grafana: <https://github.com/kubernetes-sigs/kubebuilder/tree/v3.7.0/pkg/plugins/optional/grafana/v1alpha>
diff --git a/docs/book/src/plugins/plugins.md b/docs/book/src/plugins/plugins.md
@@ -19,7 +19,7 @@ This section details how to extend Kubebuilder and create your plugins following
 <aside class="note">
 <h1>Note</h1>
 
-You can check the existing design proposal docs at [Extensible CLI and Scaffolding Plugins: phase 1][plugins-phase1-design-doc] and [Extensible CLI and Scaffolding Plugins: phase 1.5][plugins-phase1-design-doc-1.5] to know more on what is provided by Kubebuilder CLI and API currently. 
+You can check the existing design proposal docs at [Extensible CLI and Scaffolding Plugins: phase 1][plugins-phase1-design-doc] and [Extensible CLI and Scaffolding Plugins: phase 1.5][plugins-phase1-design-doc-1.5] to know more on what is provided by Kubebuilder CLI and API currently.
 
 </aside>
 
@@ -30,12 +30,11 @@ To know more about Kubebuilder's future vision of the Plugins architecture, see
 
 </aside>
 
-  - [Extending the CLI and Scaffolds](extending-cli.md)
-  - [Creating your own plugins](creating-plugins.md)
+- [Extending the CLI and Scaffolds](extending-cli.md)
+- [Creating your own plugins](creating-plugins.md)
+- [Testing your plugins](testing-plugins.md)
 
- [plugins-phase1-design-doc]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1.md
- [plugins-phase1-design-doc-1.5]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1-5.md
- [extending-cli]: extending-cli.md
-
-
- [section-future-vision-plugins]: https://book.kubebuilder.io/plugins/creating-plugins.html#future-vision-for-kubebuilder-plugins
+[plugins-phase1-design-doc]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1.md
+[plugins-phase1-design-doc-1.5]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1-5.md
+[extending-cli]: extending-cli.md
+[section-future-vision-plugins]: https://book.kubebuilder.io/plugins/creating-plugins.html#future-vision-for-kubebuilder-plugins