-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Chris Hein <me@chrishein.com>
- Loading branch information
1 parent
95c0327
commit 7aae5bd
Showing
2 changed files
with
322 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,322 @@ | ||
# ComponentConfig Controller Runtime Support | ||
Author: @christopherhein | ||
|
||
Last Updated on: 03/02/2020 | ||
|
||
## Table of Contents | ||
|
||
<!--ts--> | ||
* [ComponentConfig Controller Runtime Support](#componentconfig-controller-runtime-support) | ||
* [Table of Contents](#table-of-contents) | ||
* [Summary](#summary) | ||
* [Motivation](#motivation) | ||
* [Links to Open Issues](#links-to-open-issues) | ||
* [Goals](#goals) | ||
* [Non-Goals/Future Work](#non-goalsfuture-work) | ||
* [Proposal](#proposal) | ||
* [ComponentConfig Load Order](#componentconfig-load-order) | ||
* [Default ComponentConfig Type](#default-componentconfig-type) | ||
* [Caveats](#caveats) | ||
* [Kubebuilder Scaffolding Example](#kubebuilder-scaffolding-example) | ||
* [User Stories](#user-stories) | ||
* [Controller Author with controller-runtime](#controller-author-with-controller-runtime) | ||
* [Controller Author with kubebuilder (tbd proposal for kubebuilder)](#controller-author-with-kubebuilder-tbd-proposal-for-kubebuilder) | ||
* [Controller User without modifications to config](#controller-user-without-modifications-to-config) | ||
* [Controller User with modifications to config](#controller-user-with-modifications-to-config) | ||
* [Risks and Mitigations](#risks-and-mitigations) | ||
* [Alternatives](#alternatives) | ||
* [Implementation History](#implementation-history) | ||
|
||
<!--te--> | ||
|
||
## Summary | ||
|
||
Currently controllers that use `controller-runtime` need to configure the `ctrl.Manager` by using flags or hardcoding values into the initialization methods. Core Kubernetes has started to move away from using flags as a mechanism for configuring components and standardized along the pattern of [`ComponentConfig` or Versioned Component Configuration Files](https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/edit). This proposal is to bring `ComponentConfig` patterns into `controller-runtime` to allow controller authors to make `go` types backed by `apimachinery` to unmarshal and configure the `ctrl.Manager` reducing the flags and allowing code based tools to easily configure controllers instead of requiring them to mutate CLI args. | ||
|
||
|
||
## Motivation | ||
|
||
This change is important because: | ||
- it will help make it easier for controllers to be configured by other machine processes | ||
- it will reduce the required flags required to start a controller | ||
- allow for more configuration types which flags don't natively support | ||
- allow using and upgrading older configurations avoiding breaking changes in flags | ||
|
||
### Links to Open Issues | ||
|
||
- [#518 Provide a ComponentConfig to tweak the Manager](https://github.com/kubernetes-sigs/controller-runtime/issues/518) | ||
- [#207 Reduce command line flag boilerplate](https://github.com/kubernetes-sigs/controller-runtime/issues/207) | ||
- [#722 Implement ComponentConfig by default & stop using (most) flags](https://github.com/kubernetes-sigs/kubebuilder/issues/722) | ||
|
||
### Goals | ||
|
||
- Provide an interface for pulling configuration data out of exposed `ComponentConfig` types (see below for implementation) | ||
- Provide a new `ctrl.NewFromComponentConfig()` function for initializing a manager | ||
- Provide a `DefaultControllerConfig` to make the switch easier | ||
|
||
### Non-Goals/Future Work | ||
|
||
- `kubebuilder` implementation and design in another PR | ||
- Changing the default `controller-runtime` implementation | ||
- Dynamically reloading ComponentConfig object | ||
- Providing `flags` interface and overrides | ||
|
||
## Proposal | ||
|
||
The `ctrl.Manager` _SHOULD_ support loading configurations from `ComponentConfig` like objects. | ||
An interface for that object with getters for the specific configuration parameters is created to bridge existing patterns. | ||
|
||
Without breaking the current `ctrl.NewManager` which uses an exported `ctrl.Options{}` the `manager.go` can expose a new func, `NewFromComponentConfig()` this would be able to loop through the getters to hydrate an internal `ctrl.Options{}` and pass that into `New()`. | ||
|
||
File: _`pkg/manager/manager.go`_ | ||
```golang | ||
// ManagerConfiguration defines what the ComponentConfig object for ControllerRuntime needs to support | ||
type ManagerConfiguration interface { | ||
GetSyncPeriod() *time.Duration | ||
|
||
GetLeaderElection() bool | ||
GetLeaderElectionNamespace() string | ||
GetLeaderElectionID() string | ||
|
||
GetLeaseDuration() *time.Duration | ||
GetRenewDeadline() *time.Duration | ||
GetRetryPeriod() *time.Duration | ||
|
||
GetNamespace() string | ||
GetMetricsBindAddress() string | ||
GetHealthProbeBindAddress() string | ||
|
||
GetReadinessEndpointName() string | ||
GetLivenessEndpointName() string | ||
|
||
GetPort() int | ||
GetHost() string | ||
|
||
GetCertDir() string | ||
} | ||
|
||
func NewFromComponentConfig(config *rest.Config, scheme *runtime.Scheme,filename string, managerconfig ManagerConfiguration) (Manager, error) { | ||
codecs := serializer.NewCodecFactory(scheme) | ||
if err := decodeComponentConfigFileInto(codecs, filename, componentconfig); err != nil { | ||
|
||
} | ||
options := Options{} | ||
|
||
if scheme != nil { | ||
options.Scheme = scheme | ||
} | ||
|
||
// Loop through getters | ||
if managerconfig.GetLeaderElection() { | ||
managerconfig.GetLeaderElection() | ||
} | ||
// ... | ||
|
||
return New(config, options) | ||
} | ||
``` | ||
|
||
#### ComponentConfig Load Order | ||
|
||
![ComponentConfig Load Order](/designs/images/component-config-load.png) | ||
|
||
#### Default ComponentConfig Type | ||
|
||
To enable `controller-runtime` to have a default `ComponentConfig` struct which can be used instead of requiring each controller or extension to build it's own `ComponentConfig` type, we can create a `DefaultControllerConfiguration` type which can exist in `pkg/api/v1alpha1/types.go`. This will allow the controller authors to use this before needing to implement their own type with additional configs. | ||
|
||
```golang | ||
package v1alpha1 | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
configv1alpha1 "sigs.k8s.io/component-base/config/v1alpha1" | ||
) | ||
|
||
// DefaultControllerConfigurationSpec defines the desired state of DefaultControllerConfiguration | ||
type DefaultControllerConfigurationSpec struct { | ||
SyncPeriod *time.Duration `json:"syncPeriod,omitempty"` | ||
|
||
LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"` | ||
|
||
MetricsBindAddress string `json:"metricsBindAddress,omitempty"` | ||
|
||
Health DefaultControllerConfigurationHealth `json:"health,omitempty"` | ||
|
||
Port *int `json:"port,omitempty"` | ||
Host string `json:"host,omitempty"` | ||
|
||
CertDir string `json:"certDir,omitempty"` | ||
} | ||
|
||
// DefaultControllerConfigurationHealth defines the health configs | ||
type DefaultControllerConfigurationHealth struct { | ||
HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"` | ||
|
||
ReadinessEndpointName string `json:"readinessEndpointName,omitempty"` | ||
LivenessEndpointName string `json:"livenessEndpointName,omitempty"` | ||
} | ||
|
||
// DefaultControllerConfiguration is the Schema for the DefaultControllerConfigurations API | ||
type DefaultControllerConfiguration struct { | ||
metav1.TypeMeta `json:",inline"` | ||
|
||
Spec DefaultControllerConfigurationSpec `json:"spec,omitempty"` | ||
} | ||
``` | ||
|
||
This would allow a controller author to use this struct with any config that supports the json/yaml structure. For example a controller author could define their `Kind` as `FoobarControllerConfiguration` and have it defined as the following. | ||
|
||
```yaml | ||
# config.yaml | ||
apiVersion: somedomain.io/v1alpha1 | ||
kind: FoobarControllerConfiguration | ||
spec: | ||
port: 9443 | ||
metricsBindAddress: ":8080" | ||
leaderElection: | ||
leaderElect: false | ||
``` | ||
|
||
Given the following config and `DefaultControllerConfiguration` we'd be able to initialize the controller using the following. | ||
|
||
|
||
```golang | ||
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, &defaultv1alpha1.DefaultControllerConfiguration{}) | ||
if err != nil { | ||
// ... | ||
} | ||
``` | ||
|
||
The above example uses `configname` which is the name of the file to load the configuration from and uses `scheme` to get the specific serializer, eg `serializer.NewCodecFactory(scheme)`. This will allow the configuration to be unmarshalled into the `runtime.Object` type and passed into the | ||
`ctrl.NewManagerFromComponentConfig()` as a `ManagerConfiguration` interface. | ||
|
||
##### Caveats | ||
|
||
> ⚠️ Using `DecodeComponentConfigFileInto` does not support the overrides for flags, this is something that is left up to the controller author since they could be using many different flagging interfaces. eg [`flag`](https://golang.org/pkg/flag/), [`pflag`](https://godoc.org/github.com/spf13/pflag), [`flagnum`](https://godoc.org/github.com/luci/luci-go/common/flag/flagenum) and `controller-runtime` should be agnostic to the CLI implementation. | ||
#### Kubebuilder Scaffolding Example | ||
|
||
Within a separate design (link once created) this will require controller authors to generate a type that implements the `ManagerConfiguration` interface. The following is a sample of what this looks like: | ||
|
||
```golang | ||
package config | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
configv1alpha1 "sigs.k8s.io/component-base/config/v1alpha1" | ||
) | ||
|
||
type ControllerNameConfigurationSpec struct { | ||
SyncPeriod *time.Duration `json:"syncPeriod,omitempty"` | ||
|
||
LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"` | ||
|
||
MetricsBindAddress string `json:"metricsBindAddress,omitempty"` | ||
|
||
Health ControllerNameConfigurationHealth | ||
|
||
Port int `json:"port,omitempty"` | ||
Host string `json:"host,omitempty"` | ||
|
||
CertDir string `json:"certDir,omitempty"` | ||
} | ||
|
||
type ControllerNameConfigurationHealth struct { | ||
HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"` | ||
|
||
ReadinessEndpointName string `json:"readinessEndpointName,omitempty"` | ||
LivenessEndpointName string `json:"livenessEndpointName,omitempty"` | ||
} | ||
|
||
type ControllerNameConfiguration struct { | ||
metav1.TypeMeta | ||
|
||
Spec ControllerNameConfigurationSpec `json:"spec"` | ||
} | ||
``` | ||
|
||
With the above `struct` we would need to implement the getters to satisfy the `ManagerConfiguration` interface, these could be automatically generated when you `init` or `create componentconfig` from the `kubebuilder` CLI. This could be something as simple as the following. | ||
|
||
```golang | ||
package config | ||
|
||
import ( | ||
"time" | ||
) | ||
|
||
func (in *ControllerNameConfiguration) GetSyncPeriod() *time.Duration {} | ||
func (in *ControllerNameConfiguration) GetLeaderElection() bool {} | ||
func (in *ControllerNameConfiguration) GetLeaderElectionNamespace() string {} | ||
func (in *ControllerNameConfiguration) GetLeaderElectionID() string {} | ||
|
||
func (in *ControllerNameConfiguration) GetLeaseDuration() *time.Duration {} | ||
func (in *ControllerNameConfiguration) GetRenewDeadline() *time.Duration {} | ||
func (in *ControllerNameConfiguration) GetRetryPeriod() *time.Duration {} | ||
|
||
func (in *ControllerNameConfiguration) GetNamespace() string {} | ||
func (in *ControllerNameConfiguration) GetMetricsBindAddress() string {} | ||
func (in *ControllerNameConfiguration) GetHealthProbeBindAddress() string {} | ||
|
||
func (in *ControllerNameConfiguration) GetReadinessEndpointName() string {} | ||
func (in *ControllerNameConfiguration) GetLivenessEndpointName() string {} | ||
|
||
func (in *ControllerNameConfiguration) GetPort() int {} | ||
func (in *ControllerNameConfiguration) GetHost() string {} | ||
|
||
func (in *ControllerNameConfiguration) GetCertDir() string {} | ||
``` | ||
|
||
Besides the implementation of the `ComponentConfig` The controller author as it stands would also need to implement the unmarshalling of the `ConfigMap` into the `ComponentConfig`, for this `controller-runtime` could expose helper methods to load a file from disk, unmarshal to the struct and pass the pointer into the `NewFromComponentConfig()` to return the `ctrl.Manager` | ||
|
||
## User Stories | ||
|
||
### Controller Author with `controller-runtime` | ||
|
||
- Implement `ComponentConfig` type | ||
- Implement `ManagerConfiguration` interface for `ComponentConfig` object | ||
- Set up `ConfigMap` unmarshalling into `ComponentConfig` type | ||
- Initialize `ctrl.Manager` with `NewFromComponentConfig` | ||
- Build custom controller as usual | ||
|
||
### Controller Author with `kubebuilder` (tbd proposal for `kubebuilder`) | ||
|
||
- Initialize `kubebuilder` project using `--component-config-name=XYZConfiguration` | ||
- Build custom controller as usual | ||
|
||
### Controller User without modifications to config | ||
|
||
_Provided that the controller provides manifests_ | ||
|
||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
### Controller User with modifications to config | ||
|
||
- _Following from previous example without changes_ | ||
- Create a new `ConfigMap` for changes | ||
- Modify the `controller-runtime` pod to use the new `ConfigMap` | ||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
|
||
## Risks and Mitigations | ||
|
||
- Given that this isn't changing the core Manager initialization for `controller-runtime` it's fairly low risk | ||
|
||
## Alternatives | ||
|
||
* `NewFromComponentConfig()` could load the object from disk based on the file name and hydrate the `ComponentConfig` type. | ||
|
||
## Implementation History | ||
|
||
- [x] 02/19/2020: Proposed idea in an issue or [community meeting] | ||
- [x] 02/24/2020: Proposal submitted to `controller-runtime` | ||
- [x] 03/02/2020: Updated with default `DefaultControllerConfiguration` | ||
|
||
|
||
<!-- Links --> | ||
[community meeting]: https://docs.google.com/document/d/1Ih-2cgg1bUrLwLVTB9tADlPcVdgnuMNBGbUl4D-0TIk |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.