diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index 6ddbcdbab51..cb389867449 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -106,6 +106,10 @@ - [completion](./reference/completion.md) - [Artifacts](./reference/artifacts.md) - [Platform Support](./reference/platform.md) + + - [Sub-Module Layouts](./reference/submodule-layouts.md) + - [Using an external Type / API](./reference/using_an_external_type.md) + - [Configuring EnvTest](./reference/envtest.md) - [Metrics](./reference/metrics.md) diff --git a/docs/book/src/reference/reference.md b/docs/book/src/reference/reference.md index cbd05bb62bc..1d9026777d8 100644 --- a/docs/book/src/reference/reference.md +++ b/docs/book/src/reference/reference.md @@ -31,10 +31,12 @@ - [completion](completion.md) - [Artifacts](artifacts.md) - [Platform Support](platform.md) - - [Writing controller tests](writing-tests.md) - - [Metrics](metrics.md) + - [Sub-Module Layouts](submodule-layouts.md) + - [Using an external Type / API](using_an_external_type.md) + + - [Metrics](metrics.md) - [Reference](metrics-reference.md) - [Makefile Helpers](makefile-helpers.md) - - [CLI plugins](../plugins/cli-plugins.md) + - [CLI plugins](../plugins/plugins.md) diff --git a/docs/book/src/reference/submodule-layouts.md b/docs/book/src/reference/submodule-layouts.md new file mode 100644 index 00000000000..69fd1df8f92 --- /dev/null +++ b/docs/book/src/reference/submodule-layouts.md @@ -0,0 +1,248 @@ +# Sub-Module Layouts + +This part describes how to modify a scaffolded project for use with multiple `go.mod` files for APIs and Controllers. + +Sub-Module Layouts (in a way you could call them a special form of [Monorepo's][monorepo]) are a special use case and can help in scenarios that involve reuse of APIs without introducing indirect dependencies that should not be available in the project consuming the API externally. + + + +## Overview + +Separate `go.mod` modules for APIs and Controllers can help for the following cases: + +- There is an enterprise version of an operator available that wants to reuse APIs from the Community Version +- There are many (possibly external) modules depending on the API and you want to have a more strict separation of transitive dependencies +- If you want to reduce impact of transitive dependencies on your API being included in other projects +- If you are looking to separately manage the lifecycle of your API release process from your controller release process. +- If you are looking to modularize your codebase without splitting your code between multiple repositories. + +They introduce however multiple caveats into typical projects which is one of the main factors that makes them hard to recommend in a generic use-case or plugin: + +- Multiple `go.mod` modules are not recommended as a go best practice and [multiple modules are mostly discouraged][multi-module-repositories] +- There is always the possibility to extract your APIs into a new repository and arguably also have more control over the release process in a project spanning multiple repos relying on the same API types. +- It requires at least one [replace directive][replace-directives] either through `go.work` which is at least 2 more files plus an environment variable for build environments without GO_WORK or through `go.mod` replace, which has to be manually dropped and added for every release. + + + +## Adjusting your Project + +For a proper Sub-Module layout, we will use the generated APIs as a starting point. + +For the steps below, we will assume you created your project in your `GOPATH` with + +```shell +kubebuilder init +``` + +and created an API & controller with + +```shell +kubebuilder create api --group operator --version v1alpha1 --kind Sample --resource --controller --make +``` + +### Creating a second module for your API + +Now that we have a base layout in place, we will enable you for multiple modules. + +1. Navigate to `api/v1alpha1` +2. Run `go mod init` to create a new submodule +3. Run `go mod tidy` to resolve the dependencies + +Your api go.mod file could now look like this: + +```go.mod +module YOUR_GO_PATH/test-operator/api/v1alpha1 + +go 1.21.0 + +require ( + k8s.io/apimachinery v0.28.4 + sigs.k8s.io/controller-runtime v0.16.3 +) + +require ( + github.com/go-logr/logr v1.2.4 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/google/gofuzz v1.2.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + golang.org/x/net v0.17.0 // indirect + golang.org/x/text v0.13.0 // indirect + gopkg.in/inf.v0 v0.9.1 // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect + k8s.io/klog/v2 v2.100.1 // indirect + k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 // indirect + sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect + sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect +) +``` + +As you can see it only includes apimachinery and controller-runtime as dependencies and any dependencies you have +declared in your controller are not taken over into the indirect imports. + +### Using replace directives for development + +When trying to resolve your main module in the root folder of the operator, you will notice an error if you use a VCS path: + +```shell +go mod tidy +go: finding module for package YOUR_GO_PATH/test-operator/api/v1alpha1 +YOUR_GO_PATH/test-operator imports + YOUR_GO_PATH/test-operator/api/v1alpha1: cannot find module providing package YOUR_GO_PATH/test-operator/api/v1alpha1: module YOUR_GO_PATH/test-operator/api/v1alpha1: git ls-remote -q origin in LOCALVCSPATH: exit status 128: + remote: Repository not found. + fatal: repository 'https://YOUR_GO_PATH/test-operator/' not found +``` + +The reason for this is that you may have not pushed your modules into the VCS yet and resolving the main module will fail as it can no longer +directly access the API types as a package but only as a module. + +To solve this issue, we will have to tell the go tooling to properly `replace` the API module with a local reference to your path. + +You can do this with 2 different approaches: go modules and go workspaces. + +#### Using go modules + +For go modules, you will edit the main `go.mod` file of your project and issue a replace directive. + +You can do this by editing the `go.mod` with +`` +```shell +go mod edit -require YOUR_GO_PATH/test-operator/api/v1alpha1@v0.0.0 # Only if you didn't already resolve the module +go mod edit -replace YOUR_GO_PATH/test-operator/api/v1alpha1@v0.0.0=./api/v1alpha1 +go mod tidy +``` + +Note that we used the placeholder version `v0.0.0` of the API Module. In case you already released your API module once, +you can use the real version as well. However this will only work if the API Module is already available in the VCS. + + + +#### Using go workspaces + +For go workspaces, you will not edit the `go.mod` files yourself, but rely on the workspace support in go. + +To initialize a workspace for your project, run `go work init` in the project root. + +Now let us include both modules in our workspace: +```shell +go work use . # This includes the main module with the controller +go work use api/v1alpha1 # This is the API submodule +go work sync +``` + +This will lead to commands such as `go run` or `go build` to respect the workspace and make sure that local resolution is used. + +You will be able to work with this locally without having to build your module. + +When using `go.work` files, it is recommended to not commit them into the repository and add them to `.gitignore`. + +```gitignore +go.work +go.work.sum +``` + +When releasing with a present `go.work` file, make sure to set the environment variable `GOWORK=off` (verifiable with `go env GOWORK`) to make sure the release process does not get impeded by a potentially commited `go.work` file. + +#### Adjusting the Dockerfile + +When building your controller image, kubebuilder by default is not able to work with multiple modules. +You will have to manually add the new API module into the download of dependencies: + +```dockerfile +# Build the manager binary +FROM golang:1.20 as builder +ARG TARGETOS +ARG TARGETARCH + +WORKDIR /workspace +# Copy the Go Modules manifests +COPY go.mod go.mod +COPY go.sum go.sum +# Copy the Go Sub-Module manifests +COPY api/v1alpha1/go.mod api/go.mod +COPY api/v1alpha1/go.sum api/go.sum +# cache deps before building and copying source so that we don't need to re-download as much +# and so that source changes don't invalidate our downloaded layer +RUN go mod download + +# Copy the go source +COPY cmd/main.go cmd/main.go +COPY api/ api/ +COPY internal/controller/ internal/controller/ + +# Build +# the GOARCH has not a default value to allow the binary be built according to the host where the command +# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO +# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore, +# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform. +RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go + +# Use distroless as minimal base image to package the manager binary +# Refer to https://github.com/GoogleContainerTools/distroless for more details +FROM gcr.io/distroless/static:nonroot +WORKDIR / +COPY --from=builder /workspace/manager . +USER 65532:65532 + +ENTRYPOINT ["/manager"] +``` + +### Creating a new API and controller release + +Because you adjusted the default layout, before releasing your first version of your operator, make sure to [familiarize yourself with mono-repo/multi-module releases][multi-module-repositories] with multiple `go.mod` files in different subdirectories. + +Assuming a single API was created, the release process could look like this: + +```sh +git commit +git tag v1.0.0 # this is your main module release +git tag api/v1.0.0 # this is your api release +go mod edit -require YOUR_GO_PATH/test-operator/api@v1.0.0 # now we depend on the api module in the main module +go mod edit -dropreplace YOUR_GO_PATH/test-operator/api/v1alpha1 # this will drop the replace directive for local development in case you use go modules, meaning the sources from the VCS will be used instead of the ones in your monorepo checked out locally. +git push origin main v1.0.0 api/v1.0.0 +``` + +After this, your modules will be available in VCS and you do not need a local replacement anymore. However if youre making local changes, +make sure to adopt your behavior with `replace` directives accordingly. + +### Reusing your extracted API module + +Whenever you want to reuse your API module with a separate kubebuilder, we will assume you follow the guide for [using an external Type](/reference/using_an_external_type.md). +When you get to the step `Edit the API files` simply import the dependency with + +```shell +go get YOUR_GO_PATH/test-operator/api@v1.0.0 +``` + +and then use it as explained in the guide. + +[monorepo]: https://en.wikipedia.org/wiki/Monorepo +[replace-directives]: https://go.dev/ref/mod#go-mod-file-replace +[multi-module-repositories]: https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories \ No newline at end of file diff --git a/docs/book/src/reference/using_an_external_type.md b/docs/book/src/reference/using_an_external_type.md new file mode 100644 index 00000000000..14fcd522048 --- /dev/null +++ b/docs/book/src/reference/using_an_external_type.md @@ -0,0 +1,275 @@ +# Using an External Type + +There are several different external types that may be referenced when writing a controller. +* Custom Resource Definitions (CRDs) that are defined in the current project (such as via `kubebuilder create api`). +* Core Kubernetes Resources (e.g. Deployments or Pods). +* CRDs that are created and installed in another project. +* A custom API defined via the aggregation layer, served by an extension API server for which the primary API server acts as a proxy. + +Currently, kubebuilder handles the first two, CRDs and Core Resources, seamlessly. You must scaffold the latter two, External CRDs and APIs created via aggregation, manually. + +In order to use a Kubernetes Custom Resource that has been defined in another project +you will need to have several items of information. +* The Domain of the CR +* The Group under the Domain +* The Go import path of the CR Type definition +* The Custom Resource Type you want to depend on. + +The Domain and Group variables have been discussed in other parts of the documentation. The import path would be located in the project that installs the CR. +The Custom Resource Type is usually a Go Type of the same name as the CustomResourceDefinition in kubernetes, e.g. for a `Pod` there will be a type `Pod` in the `v1` group. +For Kubernetes Core Types, the domain can be omitted. +`` +This document uses `my` and `their` prefixes as a naming convention for repos, groups, and types to clearly distinguish between your own project and the external one you are referencing. + +In our example we will assume the following external API Type: + +`github.com/theiruser/theirproject` is another kubebuilder project on whose CRD we want to depend and extend on. +Thus, it contains a `go.mod` in its repository root. The import path for the go types would be `github.com/theiruser/theirproject/api/theirgroup/v1alpha1`. + +The Domain of the CR is `theirs.com`, the Group is `theirgroup` and the kind and go type would be `ExternalType`. + +If there is an interest to have multiple Controllers running in different Groups (e.g. because one is an owned CRD and one is an external Type), please first +reconfigure the Project to use a multi-group layout as described in the [Multi-Group documentation](../migration/multi-group.md). + +### Prerequisites + +The following guide assumes that you have already created a project using `kubebuilder init` in a directory in the GOPATH. Please reference the [Getting Started Guide](../getting-started.md) for more information. + +Note that if you did not pass `--domain` to `kubebuilder init` you will need to modify it for the individual api types as the default is `my.domain`, not `theirs.com`. +Similarly, if you intend to use your own domain, please configure your own domain with `kubebuilder init` and do not use `theirs.com for the domain. + +### Add a controller for the external Type + +Run the command `create api` to scaffold only the controller to manage the external type: + +```shell +kubebuilder create api --group --version v1alpha1 --kind --controller --resource=false +``` + +Note that the `resource` argument is set to false, as we are not attempting to create our own CustomResourceDefinition, +but instead rely on an external one. + +This will result in a `PROJECT` entry with the default domain of the `PROJECT` (`my.domain` if not specified in `kubebuilder init`). +For use of other domains, such as `theirs.com`, one will have to manually adjust the `PROJECT` file with the correct domain for the entry: + + + +file: PROJECT +``` +domain: my.domain +layout: +- go.kubebuilder.io/v4 +projectName: testkube +repo: example.com +resources: +- controller: true + domain: my.domain ## <- Replace the domain with theirs.com domain + group: mygroup + kind: ExternalType + version: v1alpha1 +version: "3" +``` + +At the same time, the generated RBAC manifests need to be adjusted: + +file: internal/controller/externaltype_controller.go +```go +// ExternalTypeReconciler reconciles a ExternalType object +type ExternalTypeReconciler struct { + client.Client + Scheme *runtime.Scheme +} + +// external types can be added like this +//+kubebuilder:rbac:groups=theirgroup.theirs.com,resources=externaltypes,verbs=get;list;watch;create;update;patch;delete +//+kubebuilder:rbac:groups=theirgroup.theirs.com,resources=externaltypes/status,verbs=get;update;patch +//+kubebuilder:rbac:groups=theirgroup.theirs.com,resources=externaltypes/finalizers,verbs=update +// core types can be added like this +//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;create;update;patch;delete +//+kubebuilder:rbac:groups=core,resources=pods/status,verbs=get;list;watch;create;update;patch;delete +//+kubebuilder:rbac:groups=core,resources=pods/finalizers,verbs=update +``` + +### Register your Types + + + +Edit the following lines to the main.go file to register the external types: + +file: cmd/main.go +```go +package apis + +import ( + theirgroupv1alpha1 "github.com/theiruser/theirproject/apis/theirgroup/v1alpha1" +) + +func init() { + utilruntime.Must(clientgoscheme.AddToScheme(scheme)) + utilruntime.Must(theirgroupv1alpha1.AddToScheme(scheme)) // this contains the external API types + //+kubebuilder:scaffold:scheme +} +``` + +## Edit the Controller `SetupWithManager` function + +### Use the correct imports for your API and uncomment the controlled resource + +file: internal/controllers/externaltype_controllers.go +```go +package controllers + +import ( + theirgroupv1alpha1 "github.com/theiruser/theirproject/apis/theirgroup/v1alpha1" +) + +//... + +// SetupWithManager sets up the controller with the Manager. +func (r *ExternalTypeReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&theirgroupv1alpha1.ExternalType{}). + Complete(r) +} + +``` + +Note that core resources may simply be imported by depending on the API's from upstream Kubernetes and do not need additional `AddToScheme` registrations: + +file: internal/controllers/externaltype_controllers.go +```go +package controllers +// contains core resources like Deployment +import ( + v1 "k8s.io/api/apps/v1" +) + + +// SetupWithManager sets up the controller with the Manager. +func (r *ExternalTypeReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&v1.Pod{}). + Complete(r) +} +``` + +### Update dependencies + +``` +go mod tidy +``` + +### Generate RBACs with updated Groups and Resources + +``` +make manifests +``` + +## Prepare for testing + +### Register your resource in the Scheme + +Edit the `CRDDirectoryPaths` in your test suite and add the correct `AddToScheme` entry during suite initialization: + +file: internal/controllers/suite_test.go +```go +package controller + +import ( + "fmt" + "path/filepath" + "runtime" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + logf "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + //+kubebuilder:scaffold:imports + theirgroupv1alpha1 "github.com/theiruser/theirproject/apis/theirgroup/v1alpha1" +) + +var cfg *rest.Config +var k8sClient client.Client +var testEnv *envtest.Environment + +func TestControllers(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Controller Suite") +} + + +var _ = BeforeSuite(func() { + //... + By("bootstrapping test environment") + testEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{ + // if you are using vendoring and rely on a kubebuilder based project, you can simply rely on the vendored config directory + filepath.Join("..", "..", "..", "vendor", "github.com", "theiruser", "theirproject", "config", "crds"), + // otherwise you can simply download the CRD from any source and place it within the config/crd/bases directory, + filepath.Join("..", "..", "config", "crd", "bases"), + }, + ErrorIfCRDPathMissing: false, + + // The BinaryAssetsDirectory is only required if you want to run the tests directly + // without call the makefile target test. If not informed it will look for the + // default path defined in controller-runtime which is /usr/local/kubebuilder/. + // Note that you must have the required binaries setup under the bin directory to perform + // the tests directly. When we run make test it will be setup and used automatically. + BinaryAssetsDirectory: filepath.Join("..", "..", "bin", "k8s", + fmt.Sprintf("1.28.3-%s-%s", runtime.GOOS, runtime.GOARCH)), + } + + var err error + // cfg is defined in this file globally. + cfg, err = testEnv.Start() + Expect(err).NotTo(HaveOccurred()) + Expect(cfg).NotTo(BeNil()) + + //+kubebuilder:scaffold:scheme + Expect(theirgroupv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + + k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).NotTo(HaveOccurred()) + Expect(k8sClient).NotTo(BeNil()) + + +}) + +``` + +### Verifying API Availability in the Cluster + +Since we are now using external types, you will now have to rely on them being installed into the cluster. +If the APIs are not available at the time the manager starts, all informers listening to the non-available types +will fail, causing the manager to exit with an error similar to + +``` +failed to get informer from cache {"error": "Timeout: failed waiting for *v1alpha1.ExternalType Informer to sync"} +``` + +This will signal that the API Server is not yet ready to serve the external types. + +## Helpful Tips + +### Locate your domain and group variables + +The following kubectl commands may be useful + +```shell +kubectl api-resources --verbs=list -o name +kubectl api-resources --verbs=list -o name | grep my.domain +``` + diff --git a/docs/using_an_external_type.md b/docs/using_an_external_type.md deleted file mode 100644 index 13f0044d500..00000000000 --- a/docs/using_an_external_type.md +++ /dev/null @@ -1,148 +0,0 @@ -# Using an External Type - - -# Introduction - -There are several different external types that may be referenced when writing a controller. -* A Custom Resource Definition (CRD) that is defined in the current project `kubebuilder create api`. -* A Core Kubernetes Resources eg. `kubebuilder create api --group apps --version v1 --kind Deployment`. -* A CRD that is created and installed in another project. -* A CR defined via an API Aggregation (AA). Aggregated APIs are subordinate APIServers that sit behind the primary API server, which acts as a proxy. - -Currently Kubebuilder handles the first two, CRDs and Core Resources, seamlessly. External CRDs and CRs created via Aggregation must be scaffolded manually. - -In order to use a Kubernetes Custom Resource that has been defined in another project -you will need to have several items of information. -* The Domain of the CR -* The Group under the Domain -* The Go import path of the CR Type definition. - -The Domain and Group variables have been discussed in other parts of the documentation. The import path would be located in the project that installs the CR. - -This document uses `my` and `their` prefixes as a naming convention for repos, groups, and types to clearly distinguish between your own project and the external one you are referencing. - -Example external API Aggregation directory structure -``` -github.com - ├── theiruser - ├── theirproject - ├── apis - ├── theirgroup - ├── doc.go - ├── install - │   ├── install.go - ├── v1alpha1 - │   ├── doc.go - │   ├── register.go - │   ├── types.go - │   ├── zz_generated.deepcopy.go -``` - -In the case above the import path would be `github.com/theiruser/theirproject/apis/theirgroup/v1alpha1` - -### Create a project - -``` -kubebuilder init --domain $APIDOMAIN --owner "MyCompany" -``` - -### Add a controller - -be sure to answer no when it asks if you would like to create an api? [Y/n] -``` -kubebuilder create api --group mygroup --version $APIVERSION --kind MyKind - -``` - -## Edit the API files. - -### Register your Types - -Edit the following file to the pkg/apis directory to append their `AddToScheme` to your `AddToSchemes`: - -file: pkg/apis/mytype_addtoscheme.go -``` -package apis - -import ( - mygroupv1alpha1 "github.com/myuser/myrepo/apis/mygroup/v1alpha1" - theirgroupv1alpha1 "github.com/theiruser/theirproject/apis/theirgroup/v1alpha1" -) - -func init() { - // Register the types with the Scheme so the components can map objects - // to GroupVersionKinds and back - AddToSchemes = append( - AddToSchemes, - mygroupv1alpha1.SchemeBuilder.AddToScheme, - theirgroupv1alpha1.SchemeBuilder.AddToScheme, - ) -} - -``` - -## Edit the Controller files - -### Use the correct imports for your API - -file: pkg/controllers/mytype_controller.go -``` -import ( - mygroupv1alpha1 "github.com/myuser/myrepo/apis/mygroup/v1alpha1" - theirgroupv1alpha1 "github.com/theiruser/theirproject/apis/theirgroup/v1alpha1" -) -``` - -### Update dependencies - -``` -dep ensure --add -``` - -## Prepare for testing - -#### Register your resource - -Edit the `CRDDirectoryPaths` in your test suite by appending the path to their CRDs: - -file pkg/controllers/my_kind_controller_suite_test.go -``` -var cfg *rest.Config - -func TestMain(m *testing.M) { - // Get a config to talk to the apiserver - t := &envtest.Environment{ - Config: cfg, - CRDDirectoryPaths: []string{ - filepath.Join("..", "..", "..", "config", "crds"), - filepath.Join("..", "..", "..", "vendor", "github.com", "theiruser", "theirproject", "config", "crds"), - }, - UseExistingCluster: true, - } - - apis.AddToScheme(scheme.Scheme) - - var err error - if cfg, err = t.Start(); err != nil { - log.Fatal(err) - } - - code := m.Run() - t.Stop() - os.Exit(code) -} - -``` - -## Helpful Tips - -### Locate your domain and group variables - -The following kubectl commands may be useful - -``` -kubectl api-resources --verbs=list -o name - -kubectl api-resources --verbs=list -o name | grep mydomain.com -``` -