Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

📖Adding ComponentConfig proposal #818

Merged
merged 1 commit into from
Apr 2, 2020
+320 −0
Merged

📖Adding ComponentConfig proposal #818

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
320 changes: 320 additions & 0 deletions designs/component-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,320 @@
# ComponentConfig Controller Runtime Support
Author: @christopherhein

Last Updated on: 03/02/2020

## Table of Contents

<!--ts-->
* [ComponentConfig Controller Runtime Support](#componentconfig-controller-runtime-support)
* [Table of Contents](#table-of-contents)
* [Summary](#summary)
* [Motivation](#motivation)
* [Links to Open Issues](#links-to-open-issues)
* [Goals](#goals)
* [Non-Goals/Future Work](#non-goalsfuture-work)
* [Proposal](#proposal)
* [ComponentConfig Load Order](#componentconfig-load-order)
* [Embeddable ComponentConfig Type](#embeddable-componentconfig-type)
* [Default ComponentConfig Type](#default-componentconfig-type)
* [Using Flags w/ ComponentConfig](#using-flags-w-componentconfig)
* [Kubebuilder Scaffolding Example](#kubebuilder-scaffolding-example)
* [User Stories](#user-stories)
* [Controller Author with controller-runtime and default type](#controller-author-with-controller-runtime-and-default-type)
* [Controller Author with controller-runtime and custom type](#controller-author-with-controller-runtime-and-custom-type)
* [Controller Author with kubebuilder (tbd proposal for kubebuilder)](#controller-author-with-kubebuilder-tbd-proposal-for-kubebuilder)
* [Controller User without modifications to config](#controller-user-without-modifications-to-config)
* [Controller User with modifications to config](#controller-user-with-modifications-to-config)
* [Risks and Mitigations](#risks-and-mitigations)
* [Alternatives](#alternatives)
* [Implementation History](#implementation-history)

<!--te-->

## Summary

Currently controllers that use `controller-runtime` need to configure the `ctrl.Manager` by using flags or hardcoding values into the initialization methods. Core Kubernetes has started to move away from using flags as a mechanism for configuring components and standardized on [`ComponentConfig` or Versioned Component Configuration Files](https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/edit). This proposal is to bring `ComponentConfig` to `controller-runtime` to allow controller authors to make `go` types backed by `apimachinery` to unmarshal and configure the `ctrl.Manager{}` reducing the flags and allowing code based tools to easily configure controllers instead of requiring them to mutate CLI args.

## Motivation

This change is important because:
- it will help make it easier for controllers to be configured by other machine processes
- it will reduce the required flags required to start a controller
- allow for configuration types which aren't natively supported by flags
- allow using and upgrading older configurations avoiding breaking changes in flags

### Links to Open Issues

- [#518 Provide a ComponentConfig to tweak the Manager](https://github.com/kubernetes-sigs/controller-runtime/issues/518)
- [#207 Reduce command line flag boilerplate](https://github.com/kubernetes-sigs/controller-runtime/issues/207)
- [#722 Implement ComponentConfig by default & stop using (most) flags](https://github.com/kubernetes-sigs/kubebuilder/issues/722)

### Goals

- Provide an interface for pulling configuration data out of exposed `ComponentConfig` types (see below for implementation)
- Provide a new `ctrl.NewFromComponentConfig()` function for initializing a manager
- Provide an embeddable `ControllerConfiguration` type for easily authoring `ComponentConfig` types
- Provide an `DefaultControllerConfig` to make the switching easier for clients

### Non-Goals/Future Work

- `kubebuilder` implementation and design in another PR
- Changing the default `controller-runtime` implementation
- Dynamically reloading `ComponentConfig` object
- Providing `flags` interface and overrides

## Proposal

The `ctrl.Manager` _SHOULD_ support loading configurations from `ComponentConfig` like objects.
An interface for that object with getters for the specific configuration parameters is created to bridge existing patterns.

Without breaking the current `ctrl.NewManager` which uses an exported `ctrl.Options{}` the `manager.go` can expose a new func, `NewFromComponentConfig()` this would be able to loop through the getters to populate an internal `ctrl.Options{}` and pass that into `New()`.

```golang
//pkg/manager/manager.go

// ManagerConfiguration defines what the ComponentConfig object for ControllerRuntime needs to support
type ManagerConfiguration interface {
christopherhein marked this conversation as resolved.
Show resolved Hide resolved
GetSyncPeriod() *time.Duration

GetLeaderElection() bool
GetLeaderElectionNamespace() string
GetLeaderElectionID() string

GetLeaseDuration() *time.Duration
GetRenewDeadline() *time.Duration
GetRetryPeriod() *time.Duration

GetNamespace() string
GetMetricsBindAddress() string
GetHealthProbeBindAddress() string

GetReadinessEndpointName() string
GetLivenessEndpointName() string

GetPort() int
GetHost() string

GetCertDir() string
}

func NewFromComponentConfig(config *rest.Config, scheme *runtime.Scheme, filename string, managerconfig ManagerConfiguration) (Manager, error) {
christopherhein marked this conversation as resolved.
Show resolved Hide resolved
codecs := serializer.NewCodecFactory(scheme)
if err := decodeComponentConfigFileInto(codecs, filename, managerconfig); err != nil {

}
options := Options{}

if scheme != nil {
options.Scheme = scheme
}

// Loop through getters
if managerconfig.GetLeaderElection() {
options.LeaderElection = managerconfig.GetLeaderElection()
}
// ...

return New(config, options)
}
```

#### ComponentConfig Load Order

![ComponentConfig Load Order](/designs/images/component-config-load.png)

#### Embeddable ComponentConfig Type
christopherhein marked this conversation as resolved.
Show resolved Hide resolved

To make this easier for Controller authors `controller-runtime` can expose a set of `config.ControllerConfiguration` type that can be embedded similar to the way that `k8s.io/apimachinery/pkg/apis/meta/v1` works for `TypeMeta` and `ObjectMeta` these could live in `pkg/api/config/v1alpha1/types.go`. See the `DefaultComponentConfig` type below for and example implementation.

```golang
// pkg/api/config/v1alpha1/types.go
package v1alpha1

import (
"time"

configv1alpha1 "k8s.io/component-base/config/v1alpha1"
)

// ControllerConfiguration defines the embedded RuntimeConfiguration for controller-runtime clients.
type ControllerConfiguration struct {
christopherhein marked this conversation as resolved.
Show resolved Hide resolved
Namespace string `json:"namespace,omitempty"`

SyncPeriod *time.Duration `json:"syncPeriod,omitempty"`

LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"`

MetricsBindAddress string `json:"metricsBindAddress,omitempty"`

Health ControllerConfigurationHealth `json:"health,omitempty"`

Port *int `json:"port,omitempty"`
Host string `json:"host,omitempty"`

CertDir string `json:"certDir,omitempty"`
}
christopherhein marked this conversation as resolved.
Show resolved Hide resolved

// ControllerConfigurationHealth defines the health configs
type ControllerConfigurationHealth struct {
HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"`

ReadinessEndpointName string `json:"readinessEndpointName,omitempty"`
LivenessEndpointName string `json:"livenessEndpointName,omitempty"`
}
```



#### Default ComponentConfig Type

To enable `controller-runtime` to have a default `ComponentConfig` struct which can be used instead of requiring each controller or extension to build it's own `ComponentConfig` type, we can create a `DefaultControllerConfiguration` type which can exist in `pkg/api/config/v1alpha1/types.go`. This will allow the controller authors to use this before needing to implement their own type with additional configs.

```golang
// pkg/api/config/v1alpha1/types.go
package v1alpha1

import (
"time"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1"
)

// DefaultControllerConfiguration is the Schema for the DefaultControllerConfigurations API
type DefaultControllerConfiguration struct {
metav1.TypeMeta `json:",inline"`

Spec configv1alpha1.ControllerConfiguration `json:"spec,omitempty"`
}
```

This would allow a controller author to use this struct with any config that supports the json/yaml structure. For example a controller author could define their `Kind` as `FoobarControllerConfiguration` and have it defined as the following.

```yaml
# config.yaml
apiVersion: config.somedomain.io/v1alpha1
kind: FoobarControllerConfiguration
christopherhein marked this conversation as resolved.
Show resolved Hide resolved
spec:
port: 9443
metricsBindAddress: ":8080"
leaderElection:
leaderElect: false
```

Given the following config and `DefaultControllerConfiguration` we'd be able to initialize the controller using the following.


```golang
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, &defaultv1alpha1.DefaultControllerConfiguration{})
if err != nil {
// ...
}
```

The above example uses `configname` which is the name of the file to load the configuration from and uses `scheme` to get the specific serializer, eg `serializer.NewCodecFactory(scheme)`. This will allow the configuration to be unmarshalled into the `runtime.Object` type and passed into the
`ctrl.NewManagerFromComponentConfig()` as a `ManagerConfiguration` interface.

#### Using Flags w/ ComponentConfig

Since this design still requires setting up the initial `ComponentConfig` type and passing in a pointer to `ctrl.NewFromComponentConfig()` if you want to allow for the use of flags, your controller can use any of the different flagging interfaces. eg [`flag`](https://golang.org/pkg/flag/), [`pflag`](https://godoc.org/github.com/spf13/pflag), [`flagnum`](https://godoc.org/github.com/luci/luci-go/common/flag/flagenum) and set values on the `ComponentConfig` type prior to passing the pointer into the `ctrl.NewFromComponentConfig()`, example below.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI for backwards compatibility flags usually need to take precedence over config (unless you can force users to use exclusively flags or config).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good callout, since we're decoding in the same object that the flags could have already been set on there isn't a way I'm aware of to not override existing values while decoding. Have any suggestions?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

save off the flags and apply them after the decode? (merge the options structs together more or less)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, that probably makes the most sense. I personally don't think this needs to be documented in the design doc, do you?

Feeling okay with the rest of this @shawn-hurley ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of ctrl.NewFromComponentConfig() that returns Manager, would it be reasonable to have ctrl.NewOptionsFromComponentConfig() that returns manager.Options. And then let callers call manager.New() themselves so that they have more flexibility in updating manager.Options after it has been populated?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally a possibility. I'd originally started with that path and ended up combining the funcs to make the new API quicker to implement.

@shawn-hurley @mtaufen @DirectXMan12 @stealthybox Do you have a preference? The options would be:

  1. Use a single function ctrl.NewFromComponentConfig() which takes the ComponentConfig type and returns a ctrl.Manager.

or

  1. Separate the funcs and make it ctrl.NewOptionsFromComponentConfig() which returns manager.Options that implementation wise would be passed into manager.New().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thought would be for the ComponentConfig-populating function to accept a pointer to a manager.Options struct so that something like this would be possible:

func main() {
   opts := manager.Options{}
   _ = flag.StringVar(&opts.Namespace, "namespace", "", "Namespace to watch.")

   ctrl.UpdateOptionsFromComponentConfig(&opts)

   // Parse flags after populating from ComponentConfig so that flags
   // take precedence
   _ = flag.Parse()
   _ = manager.New(ctrl.GetConfigOrDie(), opts)
}

Copy link
Member Author

@christopherhein christopherhein Apr 1, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd need to pass similar params into each func, EG parsing out the ComponentConfig type into the options we need to have the scheme, config filename, the component config type and the Options struct.

func main() {
   opts := manager.Options{}
   _ = flag.StringVar(&opts.Namespace, "namespace", "", "Namespace to watch.")
   _ = flag.StringVar(&filename, "config", "f", "ConmponentConfig type configuration.")

   _ = ctrl.UpdateOptionsFromComponentConfig(scheme, filename, &componentconfig, &opts)

   // Parse flags after populating from ComponentConfig so that flags
   // take precedence
   _ = flag.Parse()
   _ = manager.New(ctrl.GetConfigOrDie(), opts)
}

I think both are fair flows and we could achieve the same goals with a composite function that returns the manager or returns the options.

Does anyone have strong opinions against this? @stealthybox @johnsonj @joelanford @djzager @shawn-hurley @mtaufen


```golang
leaderElect := true

config := &defaultv1alpha1.DefaultControllerConfiguration{
Spec: configv1alpha1.ControllerConfiguration{
LeaderElection: configv1alpha1.LeaderElectionConfiguration{
LeaderElect: &leaderElect,
},
},
}
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, config)
if err != nil {
// ...
}
```

#### Kubebuilder Scaffolding Example

Within expanded in a separate design _(link once created)_ this will allow controller authors to generate a type that implements the `ManagerConfiguration` interface. The following is a sample of what this looks like:

```golang
package config

import (
"time"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1"
)

type ControllerNameConfigurationSpec struct {
christopherhein marked this conversation as resolved.
Show resolved Hide resolved
configv1alpha1.ControllerConfiguration `json:",inline"`
}

type ControllerNameConfiguration struct {
metav1.TypeMeta

Spec ControllerNameConfigurationSpec `json:"spec"`
}
```

Usage of this custom `ComponentConfig` type would require then changing the `ctrl.NewFromComponentConfig()` to use the new struct vs the `DefaultControllerConfiguration`.

## User Stories

### Controller Author with `controller-runtime` and default type

- Mount `ConfigMap`
- Initialize `ctrl.Manager` with `NewFromComponentConfig` with config name and `DefaultControllerConfiguration` type
- Build custom controller as usual

### Controller Author with `controller-runtime` and custom type

- Implement `ComponentConfig` type
- Embed `ControllerConfiguration` type
- Mount `ConfigMap`
- Initialize `ctrl.Manager` with `NewFromComponentConfig` with config name and `ComponentConfig` type
- Build custom controller as usual

### Controller Author with `kubebuilder` (tbd proposal for `kubebuilder`)

- Initialize `kubebuilder` project using `--component-config-name=XYZConfiguration`
- Build custom controller as usual

### Controller User without modifications to config

_Provided that the controller provides manifests_

- Apply the controller to the cluster
- Deploy custom resources

### Controller User with modifications to config

- _Following from previous example without changes_
- Create a new `ConfigMap` for changes
- Modify the `controller-runtime` pod to use the new `ConfigMap`
- Apply the controller to the cluster
- Deploy custom resources


## Risks and Mitigations

- Given that this isn't changing the core Manager initialization for `controller-runtime` it's fairly low risk

## Alternatives

* `NewFromComponentConfig()` could load the object from disk based on the file name and hydrate the `ComponentConfig` type.

## Implementation History

- [x] 02/19/2020: Proposed idea in an issue or [community meeting]
- [x] 02/24/2020: Proposal submitted to `controller-runtime`
- [x] 03/02/2020: Updated with default `DefaultControllerConfiguration`
- [x] 03/04/2020: Updated with embeddable `RuntimeConfig`
- [x] 03/10/2020: Updated embeddable name to `ControllerConfiguration`


<!-- Links -->
[community meeting]: https://docs.google.com/document/d/1Ih-2cgg1bUrLwLVTB9tADlPcVdgnuMNBGbUl4D-0TIk
Binary file added designs/images/component-config-load.png
View file
Open in desktop
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.