Refactor kubernetes client config initialization and fix in-cluster config #690

pdecat · 2019-11-14T14:55:16Z

This PR is an alternative to #686 that intends to:

cleanup client config initialization a bit,
make sure cross-cluster works (resolve Unauthorized after update to v1.10 using token auth #679),
make sure in-cluster config works without needing extra work like Support in-cluster configuration with service accounts #497.

pdecat · 2019-11-14T17:20:53Z

kubernetes/provider.go

 	if v, ok := d.GetOk("host"); ok {
-		cfg.Host = v.(string)
+		overrides.ClusterInfo.Server = v.(string)


This change causes a regression when passing a bare IP address instead of a URL, e.g. when using this:

provider "kubernetes" { version = "1.10.1-dev3" load_config_file = false host = data.google_container_cluster.cluster0.endpoint token = data.google_client_config.default.access_token cluster_ca_certificate = base64decode( data.google_container_cluster.cluster0.master_auth[0].cluster_ca_certificate, ) }

I believe that's because overrides.ClusterInfo.Server always expects a URL: https://github.com/kubernetes/client-go/blob/v12.0.0/tools/clientcmd/api/types.go#L68-L69

While cfg.Host is more flexible: https://github.com/kubernetes/client-go/blob/v12.0.0/rest/config.go#L53-L57

A work-around is to use this:

host = "https://${data.google_container_cluster.cluster0.endpoint}:443"

But I'm experimenting how to avoid this.

Not the proper way to do it.

Alright, overrides are processed too late to be taken into account by defaultServerUrlFor():

https://github.com/kubernetes/client-go/blob/v12.0.0/tools/clientcmd/client_config.go#L176

https://github.com/kubernetes/client-go/blob/v12.0.0/rest/config.go#L438-L439

https://github.com/kubernetes/client-go/blob/v12.0.0/rest/url_utils.go#L85

https://github.com/kubernetes/client-go/blob/v12.0.0/tools/clientcmd/client_config.go#L189

https://github.com/kubernetes/client-go/blob/v12.0.0/tools/clientcmd/client_config.go#L211

pdecat · 2019-11-16T17:01:43Z

So far, I've tested this successfully in the following situations:

A subset of acceptance tests

# make testacc TEST=./kubernetes TESTARGS='-run=TestAccKubernetesConfigMap -count=1'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./kubernetes -v -run=TestAccKubernetesConfigMap -count=1 -timeout 120m
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
=== RUN   TestAccKubernetesConfigMap_basic
--- PASS: TestAccKubernetesConfigMap_basic (1.38s)
=== RUN   TestAccKubernetesConfigMap_importBasic
--- PASS: TestAccKubernetesConfigMap_importBasic (0.57s)
=== RUN   TestAccKubernetesConfigMap_binaryData
--- PASS: TestAccKubernetesConfigMap_binaryData (0.79s)
=== RUN   TestAccKubernetesConfigMap_generatedName
--- PASS: TestAccKubernetesConfigMap_generatedName (0.49s)
=== RUN   TestAccKubernetesConfigMap_importGeneratedName
--- PASS: TestAccKubernetesConfigMap_importGeneratedName (0.57s)
PASS
ok      github.com/terraform-providers/terraform-provider-kubernetes/kubernetes 3.832s

Using kubeconfig against GKE 1.13.11-gke.5

provider "kubernetes" {
  version = "1.10.1-dev6"

  config_path = var.gke_kubeconfig_gitlab-ci
}

Using access token against GKE 1.13.11-gke.14

provider "kubernetes" {
  version = "1.10.1-dev6"

  load_config_file = false
  host             = data.google_container_cluster.cluster0.endpoint
  token            = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.cluster0.master_auth[0].cluster_ca_certificate,
  )
}

In-cluster with minikube and kubernetes 1.16

https://github.com/terraform-providers/terraform-provider-kubernetes/pull/690/files#diff-52766c12b9ab0b44de46fcc3af48a543R10-R17

Cross-cluster with minikube and kubernetes 1.16 targeting minikube and kubernetes 1.15

https://github.com/terraform-providers/terraform-provider-kubernetes/pull/690/files#diff-2841352a525f3b955007a50733253d5bR11-R18

Configurations not tested yet

username / password
client_key
exec
insecure
mixes of all of the above and kubeconfig files

alexsomesan · 2019-11-22T18:28:16Z

I take this one for a ride in CI, three major cloud platforms.

houqp · 2019-12-02T21:39:58Z

@pdecat @alexsomesan what's the state of this PR? Anything I could help to move it forward?

pdecat · 2019-12-02T22:09:53Z

At this stage, I believe it mostly needs more testing and a complete review.

…corp#497)" This reverts commit e836039.

…es to support both HTTP and HTTPS when a bare IP address is passed into the host provider field

houqp · 2020-01-16T23:54:27Z

@alexsomesan anything we could do to help move this forward?

alexsomesan · 2020-01-31T00:47:05Z

I've tested this and it works as specified. I'm a bit on the fence whether we should fail when running in-cluster unless the load_config_file is set, but the alternative of automatically picking in-cluster config would surely confuse a lot of inexperienced users. Overall I agree this is the right compromise.

On the other hand, I have one suggestion about the example based on a job. I think it should be more explicit about the fact that it's not illustrating the default in-cluster scenario and explain that external credentials are being injected as well as the external API being used. I have a feeling that in-cluster API is little known enough that this clarification will save confusion going forward.

alexsomesan · 2020-01-31T00:47:35Z

@houqp aiming to add this to the upcoming release. Likely on monday.

pdecat · 2020-01-31T08:22:54Z

I've tested this and it works as specified.

Good news 🎉

I'm a bit on the fence whether we should fail when running in-cluster unless the load_config_file is set, but the alternative of automatically picking in-cluster config would surely confuse a lot of inexperienced users. Overall I agree this is the right compromise.

What about adding explicit messages with INFO or WARN log level during provider initialization to give users hints about where the configuration came from and why in-cluster config was not tried?

On the other hand, I have one suggestion about the example based on a job. I think it should be more explicit about the fact that it's not illustrating the default in-cluster scenario and explain that external credentials are being injected as well as the external API being used. I have a feeling that in-cluster API is little known enough that this clarification will save confusion going forward.

We can indeed expand a bit on the description of the job example.

Maybe referencing some other existing doc on the matter could help, e.g.:

Feel free to add what you think is best, I'm not sure I'll have time to address this before Monday.

alexsomesan · 2020-01-31T10:01:05Z

Sounds good, @pdecat. I can take care of the changes.
I like the idea about exposing the config source with log messages.

alexsomesan · 2020-02-03T12:55:28Z

I've given it more thought and I think the examples included here all have their merits. I'd like to keep them as they are. I'll follow up with a simpler one for in-cluster in addition to this.

alexsomesan

Tested manually and in CI. Works as expected.

ghost added size/L documentation size/XL and removed size/L labels Nov 14, 2019

pdecat commented Nov 14, 2019

View reviewed changes

pdecat changed the title ~~WIP: refactor kubernetes client config initialization and fix in-cluster config~~ Refactor kubernetes client config initialization and fix in-cluster config Dec 2, 2019

alexsomesan mentioned this pull request Dec 19, 2019

Running terraform apply gives Unauthorized for Kubernetes provider deploying k8s resources in only one of two identical environments #716

Closed

pdecat added 8 commits January 15, 2020 09:40

Revert "Support in-cluster configuration with service accounts (hashi…

ee80855

…corp#497)" This reverts commit e836039.

Fix in-cluster config

72b8a2b

Fix in-cluster config

99ae368

Add in-cluster and cross cluster examples

343da70

Update in-cluster example

6f42320

Fix passing bare IP addresses in provider's host field

d44bbf8

Fix in-cluster and cross-cluster examples when not using custom builds

7fc52dd

Replicate what defaultServerUrlFor() does with config but for overrid…

bff578f

…es to support both HTTP and HTTPS when a bare IP address is passed into the host provider field

pdecat force-pushed the fix-in-cluster branch from 6457242 to bff578f Compare January 15, 2020 08:40

Log when selecting to load the client config from file

dbfb1e6

alexsomesan approved these changes Feb 3, 2020

View reviewed changes

alexsomesan merged commit ae969a7 into hashicorp:master Feb 3, 2020

alexsomesan mentioned this pull request Feb 3, 2020

Only try in-cluster configuration with service account token if no other configuration was provided #686

Closed

pdecat deleted the fix-in-cluster branch February 3, 2020 13:52

rivernews mentioned this pull request Feb 8, 2020

Unauthorized after update to v1.10 using token auth #679

Closed

Victorion mentioned this pull request Feb 10, 2020

Provider 1.11.0 fails to initialize #759

Closed

pdecat mentioned this pull request Feb 18, 2020

terraform apply didn't work sethvargo/vault-on-gke#68

Closed

pdecat restored the fix-in-cluster branch February 18, 2020 13:09

ghost locked and limited conversation to collaborators Apr 21, 2020

pdecat deleted the fix-in-cluster branch July 16, 2021 12:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor kubernetes client config initialization and fix in-cluster config #690

Refactor kubernetes client config initialization and fix in-cluster config #690

pdecat commented Nov 14, 2019 •

edited

Loading

pdecat Nov 14, 2019 •

edited

Loading

pdecat Nov 14, 2019

pdecat Nov 14, 2019

pdecat Nov 14, 2019

pdecat Nov 15, 2019

pdecat commented Nov 16, 2019

alexsomesan commented Nov 22, 2019

houqp commented Dec 2, 2019 •

edited

Loading

pdecat commented Dec 2, 2019

houqp commented Jan 16, 2020

alexsomesan commented Jan 31, 2020

alexsomesan commented Jan 31, 2020

pdecat commented Jan 31, 2020

alexsomesan commented Jan 31, 2020

alexsomesan commented Feb 3, 2020

alexsomesan left a comment

Refactor kubernetes client config initialization and fix in-cluster config #690

Refactor kubernetes client config initialization and fix in-cluster config #690

Conversation

pdecat commented Nov 14, 2019 • edited Loading

pdecat Nov 14, 2019 • edited Loading

Choose a reason for hiding this comment

pdecat Nov 14, 2019

Choose a reason for hiding this comment

pdecat Nov 14, 2019

Choose a reason for hiding this comment

pdecat Nov 14, 2019

Choose a reason for hiding this comment

pdecat Nov 15, 2019

Choose a reason for hiding this comment

pdecat commented Nov 16, 2019

A subset of acceptance tests

Using kubeconfig against GKE 1.13.11-gke.5

Using access token against GKE 1.13.11-gke.14

In-cluster with minikube and kubernetes 1.16

Cross-cluster with minikube and kubernetes 1.16 targeting minikube and kubernetes 1.15

Configurations not tested yet

alexsomesan commented Nov 22, 2019

houqp commented Dec 2, 2019 • edited Loading

pdecat commented Dec 2, 2019

houqp commented Jan 16, 2020

alexsomesan commented Jan 31, 2020

alexsomesan commented Jan 31, 2020

pdecat commented Jan 31, 2020

alexsomesan commented Jan 31, 2020

alexsomesan commented Feb 3, 2020

alexsomesan left a comment

Choose a reason for hiding this comment

pdecat commented Nov 14, 2019 •

edited

Loading

pdecat Nov 14, 2019 •

edited

Loading

houqp commented Dec 2, 2019 •

edited

Loading