kubernetes_manifest might not be respecting KUBE_PROXY_URL

[kriswuollett]

I suspect that operations that implement the kubernetes_manifest is attempting to connect to non-routable IP from my local machine by not using the port-forwarded https proxy that I defined with KUBE_PROXY_URL. Creating a kubernetes_namespace with this setup works fine. My module with kubernetes_manifest resources works fine in a development cluster that has publicly accessible nodes and API endpoint without a KUBE_PROXY_URL defined.

I guess the workaround I'll have to try is converting the manifests to HCL.

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.2.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.22.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.11.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

Relevant configuration section:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.22.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.11.0"
    }
  }
  backend "gcs" {
    bucket = "MY_BUCKET"
    prefix = "MY_CLUSTER/CLUSTER_SERVICES"
  }
}

data "google_client_config" "provider" {}

provider "kubernetes" {
  host                   = var.kubernetes_host
  token                  = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(var.kubernetes_host_ca_certificate)
}

Steps to Reproduce

1. Create a Kubernetes cluster that is not accessible from the Internet like a Google Cloud cluster with private nodes and private endpoint.
2. Create a bastion host in a subnet that is authorized to access the Kubernetes endpoint. Bastion runs a http proxy like tinyproxy.
3. Open up a SSH connection to the bastion with port forwarding to the private endpoint (e.g., -L 8888:localhost:8888)
4. Run Terraform like KUBE_PROXY_URL="http://localhost:8888" terraform ....

Expected Behavior

Terraform should be able to create all kinds of kubernetes resources.

Actual Behavior

I can create a kubernetes_namespace, but when setting up third-party services like cert-manager through kubernetes_manifest resources that created with yamldecode(file(...)), I get errors like the following:

2022-06-03T16:08:49.216-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.11.0_x5: 2022-06-03T16:08:49.157-0500 [DEBUG] [InvalidClientConfiguration]: Config="&rest.Config{Host:\"https://[[PRIVATE_K8S_IP]]\", APIPath:\"\", ContentConfig:rest.ContentConfig{AcceptContentTypes:\"\", ContentType:\"\", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:(*serializer.negotiatedSerializerWrapper)(0xc001bfb5e0)}, Username:\"\", Password:\"\", BearerToken:\"--- REDACTED ---\", BearerTokenFile:\"\", Impersonate:rest.ImpersonationConfig{UserName:\"\", UID:\"\", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:\"\", CertFile:\"\", KeyFile:\"\", CAFile:\"\", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8{[[REDACTED]]}, NextProtos:[]string(nil)}, UserAgent:\"\", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}"
2022-06-03T16:08:49.217-0500 [ERROR] vertex "module.cluster-services.module.cert_manager.module.stage_1.kubernetes_manifest.file[\"manifests/ClusterRoleBinding-cert-manager-cainjector.yml\"]" error: Invalid configuration for API client

References

• similar to proxy_url value of https://localhost:<port> throws Unable to connect to the server: proxyconnect tcp: tls: first record does not look like a TLS handshake #1653, but it is clear that using the proxy works since creating and refreshing kubernetes_namespace resources works fine
• bastion discussion in Support for running port-forward while Terraform is operating #812

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[kriswuollett]

This may the case that I'm still using a data source to configure the provider since this has to occur during the plan stage? Will close as a duplicate if so once I get a chance to check.

[kriswuollett]

I'm still encountering the issue specifically with kubernetes_manifest resources. Existing resources in state like kubernetes_namespace that are already in Terraform state refresh okay before the error occurs. I changed my Kubernetes provider config to the following to avoid issues with provider dependencies during the planning stage:

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = var.kubernetes_context
}

[geNAZt]

We also hit this issue with basically the same setup. We also use a GKE private cluster and IAP tunnels to SSH on a bastion with tinyproxy on it. It works for everything non kubernetes_manifest related. But when it tries to apply kubernetes_manifest it tries to directly talk to the host URL and not the configured KUBE_PROXY_URL.

[mikeroll]

Just encountered this too. The curious thing is, while KUBE_PROXY_URL doesn't take effect for kubernetes_manifest, proxy_url in the provider configuration block does.
Judging by way the code is structured, kubernetes_manifest seems to use its own sub-provider with a separate configuration code path, so it must be somehow out of sync with that of the main provider.

Lookups for both proxy_url and KUBE_PROXY_URL go together so I have a hard time pinpointing the issue.

terraform-provider-kubernetes/manifest/provider/configure.go

Lines 492 to 508 in 73e2558

	var proxyURL string
	if !providerConfig["proxy_url"].IsNull() && providerConfig["proxy_url"].IsKnown() {
	err = providerConfig["proxy_url"].As(&proxyURL)
	if err != nil {
	// invalid attribute type - this shouldn't happen, bail out for now
	response.Diagnostics = append(response.Diagnostics, &tfprotov5.Diagnostic{
	Severity: tfprotov5.DiagnosticSeverityError,
	Summary: "Provider configuration: failed to assert type of 'proxy_url' value",
	Detail: err.Error(),
	})
	return response, nil
	}
	overrides.ClusterDefaults.ProxyURL = proxyURL
	}
	if proxyUrl, ok := os.LookupEnv("KUBE_PROXY_URL"); ok && proxyUrl != "" {
	overrides.ClusterDefaults.ProxyURL = proxyURL
	}

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[dimisjim]

Same issue here

We had to instead use the HTTP_PROXY env var

[alexsomesan]

This turned out to be caused by an embarrassing typo. Fix in above mentioned PR.

🤦