Error: Failed to construct REST client on kubernetes_manifest resource

[msfidelis]

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.1.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.21.0
+ provider registry.terraform.io/hashicorp/helm v2.6.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.12.0
+ provider registry.terraform.io/hashicorp/tls v2.2.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "karpenter" {

    computed_fields = ["spec.requirements"]

    manifest = {
        apiVersion = "karpenter.sh/v1alpha5"
        kind       = "Provisioner"

        metadata = {
            name = var.cluster_name
        }

        spec = {
            requirements = [
                {
                    key         = "karpenter.k8s.aws/instance-family"
                    operator    = "In"
                    values      = [
                        "c5", "r5"
                    ]
                },
                {
                    key         = "karpenter.sh/capacity-type"
                    operator    = "In"
                    values      = [
                        "spot"
                    ]
                },
                {
                    key         = "karpenter.k8s.aws/instance-size"
                    operator    = "In"
                    values      = [
                        "large",
                        "xlarge",
                        "2xlarge"
                    ]
                }           
            ]
            limits = {
                resources = {
                    cpu     = 100
                    memory  = "4000Gi"
                }
            }
            providerRef = {
                name    = var.cluster_name
            }

        }
    }


    depends_on = [
      helm_release.karpenter
    ]
}

Debug Output

https://gist.github.com/msfidelis/a85e6ec596ba4c762d8f3d3b76fa3aac

Steps to Reproduce

terraform apply --auto-approve 

Expected Behavior

The resource should respect the provider configuration before construct the client, like the other kubernetes provider resources.

Actual Behavior

│ Error: Failed to construct REST client
│
│   with kubernetes_manifest.karpenter[0],
│   on helm_karpenter.tf line 45, in resource "kubernetes_manifest" "karpenter":
│   45: resource "kubernetes_manifest" "karpenter" {
│
│ cannot create REST client: no client config

[alexsomesan]

That error message is saying that the client configuration to the K8s API is missing (or improperly configured).

Can you please share your provider "kubernetes" {...} block. Also, if the attributes are set from other resources / modules, please include where these are coming from as well.

[msfidelis]

I think this issue don’t sound’s like a misconfig because another kubernetes* resources works fine.

provider "helm" {
  kubernetes {
    host                   =  aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate =  base64decode(aws_eks_cluster.eks_cluster.certificate_authority.0.data)
    token                  =  data.aws_eks_cluster_auth.default.token 
  }
}

provider "kubernetes" {
  host                   =  aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate =  base64decode(aws_eks_cluster.eks_cluster.certificate_authority.0.data)
  token                  =  data.aws_eks_cluster_auth.default.token 
}

[jrsdav]

@msfidelis Have you tried a different provider version?

I have observed very odd behavior with the Kubernetes provider in versions 2.12.X similar to yours (and a few different ones); a commonality I'm seeing between your config and mine is our providers are configured to connect to the control plane the same way.

In all my cases, pinning the provider to an older version has fixed my issues. Definitely not ideal, but I'd be interested to hear if other users are experiencing similar problems.

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.11.0"
    }
  }
}

[williamohara]

I am getting it too - this was also an issue with their beta version too - ref - hashicorp/terraform-provider-kubernetes-alpha#199

I tried @jrsdav 's solution - but it didn't work for me. Could it have something to do with the cloud providers we are using - i am using azure aks... and I set that cluster up using azurerm_kubernetes_cluster from azurerm - i see that @msfidelis is using aws. How did yoy deploy your cluster @jrsdav ? Did you set it up in the same terraform.

here is my provider setup

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = ">= 3.14.0"
    }
   #.... other stuff not relevant to conversation...#
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "~> 2.11.0"
    }
  }
  
}

provider "kubernetes" {
   host = azurerm_kubernetes_cluster.core_cluster.kube_config[0].host
  username = azurerm_kubernetes_cluster.core_cluster.kube_config[0].username
  password = azurerm_kubernetes_cluster.core_cluster.kube_config[0].password
  client_certificate = base64decode(azurerm_kubernetes_cluster.core_cluster.kube_config[0].client_certificate)
  client_key = base64decode(azurerm_kubernetes_cluster.core_cluster.kube_config[0].client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.core_cluster.kube_config[0].cluster_ca_certificate)
}

[williamohara]

just reread the doc...
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest#before-you-use-this-resource

it says ...

This resource requires API access during planning time. This means the cluster has to be accessible at plan time and thus cannot be created in the same apply operation. We recommend only using this resource for custom resources or resources not yet fully supported by the provider.

I need to maintain two terraform plans one to set up the cluster - the other to throw my resources into it

[etiennnr]

FYI same error here

My entire code :

terraform {
  required_version = ">= 1.0"

  required_providers {
    kubectl = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.12.1"
    }
  }
}

provider "kubernetes" {
  config_path    = "./kubeconfig.yaml"
  config_context = "my_context_name" # redacted
}

resource "kubernetes_manifest" "some_manifest" {
  manifest = yamldecode(file("manifest.yaml"))
}

tf init works as expected
But for tf plan...

❯ tf plan
╷
│ Error: Failed to construct REST client
│ 
│   with kubernetes_manifest.some_manifest,
│   on main2.tf line 17, in resource "kubernetes_manifest" "some_manifest":
│   17: resource "kubernetes_manifest" "some_manifest" {
│ 
│ cannot create REST client: no client config

❯ tf version
Terraform v1.1.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/kubernetes v2.11.0

Your version of Terraform is out of date! The latest version
is 1.2.5. You can update by downloading from https://www.terraform.io/downloads.html

[alexsomesan]

@williamohara Nailed the problem here. The kubernetes_manifest resource needs access to the API server of the cluster during planning. This is because, in order to support CRDs in Terraform ecosystem, we need to pull the schema information for each manifest resource at runtime (during planing).

AFAICS, every person who reported seeing similar issues above, configures the attributes of the provider "kubernetes" {..} block using references to other resources' attributes. You can only do this if the referenced resource (the cluster in this case) IS ALREADY PRESENT before you start planing / applying any of the kubernetes_manifest resources. You can achieve this as simply as using the -target CLI argument to Terraform to limit the scope of the first apply to just the cluster and it's direct dependencies. Then you follow up with a second apply without a -target argument and this constructs the rest of the resources (manifest & others). You will end up with a single state file and subsequent updates no longer require this two-step approach as long as the cluster resource is present.

This limitation is stemming from Terraform itself, and the provider tries to push things as far as it can, but there is no way around needing access to schema from the API (Terraform is fundamentally a strongly-typed / schema based system).

[pfrydids]

This is also explained in the closed issue #1625

It is curious though that I can use the alternative
https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/kubectl_manifest

without issue i.e. plan without needing to make a k8s API connection .

The diff in their usage is pretty minimal

-resource "kubectl_manifest" "public_route" {
-  yaml_body = <<YAML
+resource "kubernetes_manifest" "public_route" {
+  manifest = yamldecode(<<YAML
 apiVersion: projectcontour.io/v1
 kind: HTTPProxy
 metadata:
@@ -12,4 +12,5 @@ spec:
   virtualhost:
     fqdn: ${aws_route53_record.www.name}
 YAML
+  )
 }

This is also related to:

• kubernetes_manifest does not respect provider dependencies #1732
• kubernetes_manifest does not wait for dependencies #1782
• Unable to deploy a kubernetes manifest when relying on another resource #1380

[thapakazi]

It is curious though that I can use the alternative
https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/kubectl_manifest

without issue i.e. plan without needing to make a k8s API connection .

I tested and confirm kubectl alternative you mentioned from https://github.com/gavinbunney/terraform-provider-kubectl

works just like you said @pfrydids

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[yuvipanda]

I hate making comments that are just noise and add nothing to the conversation, but doing so to placate stale-bot as this continues to be a real issue

[johnstonmatt]

I'm considering porting https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/kubectl_manifest to CDKTF if I can work out how to do that. I wish there was a less crazy way to get this done, if anyone has ideas or wants to help I'm all ears :)

edit:

if anyone is trying to deploy an ArgoCD Application CRD during the first terraform run, there is a chart that can help with that: https://artifacthub.io/packages/helm/argo/argocd-apps

by using a helm_release resource like this it is easy to deploy CRDs which don't have the restriction around the cluster being accessible during the "plan" phase, all from within CDKTF providers too!

because I'm now unblocked I won't be porting the kubectl_manifest resource to CDKTF

[ChanduReddy123]

as a workaround i have done the following

on the first run when Kubernetes is getting created

locals {
      aks_cluster_available = false
}
resource "kubernetes_manifest" "some_manifest" {
      count = local.aks_cluster_available ? 1 : 0
      manifest = yamldecode(file("manifest.yaml"))
}
resource "azurerm_kubernetes_cluster" "az_kubernetes_cluster" {
      location = ""
      name = ""
      resource_group_name = ""
      ..........
}

once the kubernetes is available we just have to switch the locals variable to true

locals {
      aks_cluster_available = true
}

[colans]

Will this be fixed by hashicorp/terraform#30937 ?

[Patafix]

This issue is still present.

[colans]

Yes, it's currently blocked by hashicorp/terraform#30937; they confirmed over there. It's actively being worked on though.