-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Chris Hein <me@chrishein.com>
- Loading branch information
1 parent
95c0327
commit 33ccd9a
Showing
2 changed files
with
312 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,312 @@ | ||
# ComponentConfig Controller Runtime Support | ||
Author: @christopherhein | ||
|
||
Last Updated on: 03/02/2020 | ||
|
||
## Table of Contents | ||
|
||
<!--ts--> | ||
* [ComponentConfig Controller Runtime Support](#componentconfig-controller-runtime-support) | ||
* [Table of Contents](#table-of-contents) | ||
* [Summary](#summary) | ||
* [Motivation](#motivation) | ||
* [Links to Open Issues](#links-to-open-issues) | ||
* [Goals](#goals) | ||
* [Non-Goals/Future Work](#non-goalsfuture-work) | ||
* [Proposal](#proposal) | ||
* [ComponentConfig Load Order](#componentconfig-load-order) | ||
* [Embeddable ComponentConfig Type](#embeddable-componentconfig-type) | ||
* [Default ComponentConfig Type](#default-componentconfig-type) | ||
* [Using Flags w/ ComponentConfig](#using-flags-w-componentconfig) | ||
* [Kubebuilder Scaffolding Example](#kubebuilder-scaffolding-example) | ||
* [User Stories](#user-stories) | ||
* [Controller Author with controller-runtime](#controller-author-with-controller-runtime) | ||
* [Controller Author with kubebuilder (tbd proposal for kubebuilder)](#controller-author-with-kubebuilder-tbd-proposal-for-kubebuilder) | ||
* [Controller User without modifications to config](#controller-user-without-modifications-to-config) | ||
* [Controller User with modifications to config](#controller-user-with-modifications-to-config) | ||
* [Risks and Mitigations](#risks-and-mitigations) | ||
* [Alternatives](#alternatives) | ||
* [Implementation History](#implementation-history) | ||
|
||
<!--te--> | ||
|
||
## Summary | ||
|
||
Currently controllers that use `controller-runtime` need to configure the `ctrl.Manager` by using flags or hardcoding values into the initialization methods. Core Kubernetes has started to move away from using flags as a mechanism for configuring components and standardized on [`ComponentConfig` or Versioned Component Configuration Files](https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/edit). This proposal is to bring `ComponentConfig` to `controller-runtime` to allow controller authors to make `go` types backed by `apimachinery` to unmarshal and configure the `ctrl.Manager{}` reducing the flags and allowing code based tools to easily configure controllers instead of requiring them to mutate CLI args. | ||
|
||
|
||
## Motivation | ||
|
||
This change is important because: | ||
- it will help make it easier for controllers to be configured by other machine processes | ||
- it will reduce the required flags required to start a controller | ||
- allow for more configuration types which flags don't natively support | ||
- allow using and upgrading older configurations avoiding breaking changes in flags | ||
|
||
### Links to Open Issues | ||
|
||
- [#518 Provide a ComponentConfig to tweak the Manager](https://github.com/kubernetes-sigs/controller-runtime/issues/518) | ||
- [#207 Reduce command line flag boilerplate](https://github.com/kubernetes-sigs/controller-runtime/issues/207) | ||
- [#722 Implement ComponentConfig by default & stop using (most) flags](https://github.com/kubernetes-sigs/kubebuilder/issues/722) | ||
|
||
### Goals | ||
|
||
- Provide an interface for pulling configuration data out of exposed `ComponentConfig` types (see below for implementation) | ||
- Provide a new `ctrl.NewFromComponentConfig()` function for initializing a manager | ||
- Provide an embeddable `RuntimeConfig` type for easily authoring `ComponentConfig` types | ||
- Provide an `DefaultControllerConfig` to make the switch easier | ||
|
||
### Non-Goals/Future Work | ||
|
||
- `kubebuilder` implementation and design in another PR | ||
- Changing the default `controller-runtime` implementation | ||
- Dynamically reloading `ComponentConfig` object | ||
- Providing `flags` interface and overrides | ||
|
||
## Proposal | ||
|
||
The `ctrl.Manager` _SHOULD_ support loading configurations from `ComponentConfig` like objects. | ||
An interface for that object with getters for the specific configuration parameters is created to bridge existing patterns. | ||
|
||
Without breaking the current `ctrl.NewManager` which uses an exported `ctrl.Options{}` the `manager.go` can expose a new func, `NewFromComponentConfig()` this would be able to loop through the getters to hydrate an internal `ctrl.Options{}` and pass that into `New()`. | ||
|
||
```golang | ||
//pkg/manager/manager.go | ||
|
||
// ManagerConfiguration defines what the ComponentConfig object for ControllerRuntime needs to support | ||
type ManagerConfiguration interface { | ||
GetSyncPeriod() *time.Duration | ||
|
||
GetLeaderElection() bool | ||
GetLeaderElectionNamespace() string | ||
GetLeaderElectionID() string | ||
|
||
GetLeaseDuration() *time.Duration | ||
GetRenewDeadline() *time.Duration | ||
GetRetryPeriod() *time.Duration | ||
|
||
GetNamespace() string | ||
GetMetricsBindAddress() string | ||
GetHealthProbeBindAddress() string | ||
|
||
GetReadinessEndpointName() string | ||
GetLivenessEndpointName() string | ||
|
||
GetPort() int | ||
GetHost() string | ||
|
||
GetCertDir() string | ||
} | ||
|
||
func NewFromComponentConfig(config *rest.Config, scheme *runtime.Scheme, filename string, managerconfig ManagerConfiguration) (Manager, error) { | ||
codecs := serializer.NewCodecFactory(scheme) | ||
if err := decodeComponentConfigFileInto(codecs, filename, componentconfig); err != nil { | ||
|
||
} | ||
options := Options{} | ||
|
||
if scheme != nil { | ||
options.Scheme = scheme | ||
} | ||
|
||
// Loop through getters | ||
if managerconfig.GetLeaderElection() { | ||
managerconfig.GetLeaderElection() | ||
} | ||
// ... | ||
|
||
return New(config, options) | ||
} | ||
``` | ||
|
||
#### ComponentConfig Load Order | ||
|
||
![ComponentConfig Load Order](/designs/images/component-config-load.png) | ||
|
||
#### Embeddable ComponentConfig Type | ||
|
||
To make this easier for Controller authors `controller-runtime` can expose a set of `meta.RuntimeConfig` type that can be embedded similar to the way that `k8s.io/apimachinery/pkg/apis/meta/v1` works for `TypeMeta` and `ObjectMeta` these could live in `pkg/api/meta/v1alpha1/types.go`. See the Default ComponentConfig type for and example implementation. | ||
|
||
```golang | ||
// pkg/api/meta/v1alpha1/types.go | ||
package v1alpha1 | ||
|
||
import ( | ||
"time" | ||
|
||
configv1alpha1 "k8s.io/component-base/config/v1alpha1" | ||
) | ||
|
||
// RuntimeConfig defines the embedded RuntimeConfiguration for controller-runtime clients. | ||
type RuntimeConfig struct { | ||
SyncPeriod *time.Duration `json:"syncPeriod,omitempty"` | ||
|
||
LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"` | ||
|
||
MetricsBindAddress string `json:"metricsBindAddress,omitempty"` | ||
|
||
Health DefaultControllerConfigurationHealth `json:"health,omitempty"` | ||
|
||
Port *int `json:"port,omitempty"` | ||
Host string `json:"host,omitempty"` | ||
|
||
CertDir string `json:"certDir,omitempty"` | ||
} | ||
|
||
// DefaultControllerConfigurationHealth defines the health configs | ||
type RuntimeConfigHealth struct { | ||
HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"` | ||
|
||
ReadinessEndpointName string `json:"readinessEndpointName,omitempty"` | ||
LivenessEndpointName string `json:"livenessEndpointName,omitempty"` | ||
} | ||
``` | ||
|
||
|
||
|
||
#### Default ComponentConfig Type | ||
|
||
To enable `controller-runtime` to have a default `ComponentConfig` struct which can be used instead of requiring each controller or extension to build it's own `ComponentConfig` type, we can create a `DefaultControllerConfiguration` type which can exist in `pkg/api/config/v1alpha1/types.go`. This will allow the controller authors to use this before needing to implement their own type with additional configs. | ||
|
||
```golang | ||
// pkg/api/config/v1alpha1/types.go | ||
package v1alpha1 | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
metav1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/meta/v1alpha1" | ||
) | ||
|
||
// DefaultControllerConfiguration is the Schema for the DefaultControllerConfigurations API | ||
type DefaultControllerConfiguration struct { | ||
metav1.TypeMeta `json:",inline"` | ||
|
||
Spec metav1alpha1.RuntimeConfig `json:"spec,omitempty"` | ||
} | ||
``` | ||
|
||
This would allow a controller author to use this struct with any config that supports the json/yaml structure. For example a controller author could define their `Kind` as `FoobarControllerConfiguration` and have it defined as the following. | ||
|
||
```yaml | ||
# config.yaml | ||
apiVersion: somedomain.io/v1alpha1 | ||
kind: FoobarControllerConfiguration | ||
spec: | ||
port: 9443 | ||
metricsBindAddress: ":8080" | ||
leaderElection: | ||
leaderElect: false | ||
``` | ||
|
||
Given the following config and `DefaultControllerConfiguration` we'd be able to initialize the controller using the following. | ||
|
||
|
||
```golang | ||
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, &defaultv1alpha1.DefaultControllerConfiguration{}) | ||
if err != nil { | ||
// ... | ||
} | ||
``` | ||
|
||
The above example uses `configname` which is the name of the file to load the configuration from and uses `scheme` to get the specific serializer, eg `serializer.NewCodecFactory(scheme)`. This will allow the configuration to be unmarshalled into the `runtime.Object` type and passed into the | ||
`ctrl.NewManagerFromComponentConfig()` as a `ManagerConfiguration` interface. | ||
|
||
#### Using Flags w/ ComponentConfig | ||
|
||
Since this design still requires setting up the initial `ComponentConfig` type and passing in a pointer to `ctrl.NewFromComponentConfig()` if you want to allow for the use of flags your controller can use any of the different flagging interfaces. eg [`flag`](https://golang.org/pkg/flag/), [`pflag`](https://godoc.org/github.com/spf13/pflag), [`flagnum`](https://godoc.org/github.com/luci/luci-go/common/flag/flagenum) and set values on the `ComponentConfig` type prior to passing the pointer into the `ctrl.NewFromComponentConfig()`. | ||
|
||
```golang | ||
leaderElect := true | ||
|
||
config := &defaultv1alpha1.DefaultControllerConfiguration{ | ||
Spec: metav1alpha1.RuntimeConfig{ | ||
LeaderElection: configv1alpha1.LeaderElectionConfiguration{ | ||
LeaderElect: &leaderElect, | ||
}, | ||
}, | ||
} | ||
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, ) | ||
if err != nil { | ||
// ... | ||
} | ||
``` | ||
|
||
#### Kubebuilder Scaffolding Example | ||
|
||
Within expanded in a separate design _(link once created)_ this will require controller authors to generate a type that implements the `ManagerConfiguration` interface. The following is a sample of what this looks like: | ||
|
||
```golang | ||
package config | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
configv1alpha1 "k8s.io/component-base/config/v1alpha1" | ||
metav1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/meta/v1alpha1" | ||
) | ||
|
||
type ControllerNameConfigurationSpec struct { | ||
metav1alpha1.RuntimeConfig `json:",inline"` | ||
} | ||
|
||
type ControllerNameConfiguration struct { | ||
metav1.TypeMeta | ||
|
||
Spec ControllerNameConfigurationSpec `json:"spec"` | ||
} | ||
``` | ||
|
||
Usage of this custom `ComponentConfig` type would require then changing the `ctrl.NewFromComponentConfig()` to use the new struct vs the `DefaultControllerConfiguration`. | ||
|
||
## User Stories | ||
|
||
### Controller Author with `controller-runtime` | ||
|
||
- Implement `ComponentConfig` type | ||
- Implement `ManagerConfiguration` interface for `ComponentConfig` object | ||
- Set up `ConfigMap` unmarshalling into `ComponentConfig` type | ||
- Initialize `ctrl.Manager` with `NewFromComponentConfig` | ||
- Build custom controller as usual | ||
|
||
### Controller Author with `kubebuilder` (tbd proposal for `kubebuilder`) | ||
|
||
- Initialize `kubebuilder` project using `--component-config-name=XYZConfiguration` | ||
- Build custom controller as usual | ||
|
||
### Controller User without modifications to config | ||
|
||
_Provided that the controller provides manifests_ | ||
|
||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
### Controller User with modifications to config | ||
|
||
- _Following from previous example without changes_ | ||
- Create a new `ConfigMap` for changes | ||
- Modify the `controller-runtime` pod to use the new `ConfigMap` | ||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
|
||
## Risks and Mitigations | ||
|
||
- Given that this isn't changing the core Manager initialization for `controller-runtime` it's fairly low risk | ||
|
||
## Alternatives | ||
|
||
* `NewFromComponentConfig()` could load the object from disk based on the file name and hydrate the `ComponentConfig` type. | ||
|
||
## Implementation History | ||
|
||
- [x] 02/19/2020: Proposed idea in an issue or [community meeting] | ||
- [x] 02/24/2020: Proposal submitted to `controller-runtime` | ||
- [x] 03/02/2020: Updated with default `DefaultControllerConfiguration` | ||
- [x] 03/04/2020: Updated with embeddable `RuntimeConfig` | ||
|
||
|
||
<!-- Links --> | ||
[community meeting]: https://docs.google.com/document/d/1Ih-2cgg1bUrLwLVTB9tADlPcVdgnuMNBGbUl4D-0TIk |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.