node.yaml parse error

[bryanfang]

/kind bug

What happened?
I need change ALLOWVOLUMEEXPANSION to true to enable resizing volume, looks like need upgrade csi-driver to v1.25
when I try to upgrade csi-driver from v2.20.0 to v2.25.0 by changing the version via terraform code
resource "helm_release" "aws_ebs_csi_driver" {
name = "aws-ebs-csi-driver"
namespace = "kube-system"
repository = "https://kubernetes-sigs.github.io/aws-ebs-csi-driver/"
chart = "aws-ebs-csi-driver"
version = "2.25.0"

values = [data.template_file.values.rendered]
}
but got error message:
18:20:06 Error: parse error at (aws-ebs-csi-driver/templates/node.yaml:1): unclosed action
18:20:06
18:20:06 on main.tf line 20, in resource "helm_release" "aws_ebs_csi_driver":
18:20:06 20: resource "helm_release" "aws_ebs_csi_driver" {
What you expected to happen?
the aws-ebs-csi-driver should be upgraded ti v2.25.0 without error
How to reproduce it (as minimally and precisely as possible)?
put the helm release to terraform code and run it in aws eks
Anything else we need to know?:
how to solve the issue
Environment
AWS EKS

• Kubernetes version (use kubectl version):
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:10:18Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/amd64"}
  Kustomize Version: v5.0.1
  Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.3-eks-4f4795d", GitCommit:"e77944ea667aee90c84c48a74a2a0da7a23e0508", GitTreeState:"clean", BuildDate:"2023-10-20T23:21:39Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
• Driver version:v1.25

[AndrewSirenko]

Hi @bryanfang,

This looks like a terraform config issue. As the source of the aws-ebs-csi-driver we can provide further clarification on configuring the helm chart, but we do not officially support deploying it via Terraform.

Additionally, I have tried running through the AWS EKS Blueprints for Terraform: Stateful pattern and was able to deploy the EBS CSI Driver via terraform using it. I would suggest looking at that setup for more guidance, or filing an issue on that repository. Those maintainers are the subject matter experts on deploying EKS add-ons via terraform.

[barbaluc]

Hi @bryanfang,
Did you find a solution ? I've have the same issue.

[gxpd-jjh]

Having same issue , no Terraform here.
EKS 1.25.

EDIT: Went back and tried to find when it broke -- helm-chart-aws-ebs-csi-driver-2.22.1 works fine, but then it breaks after that.

spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  project: default
  revisionHistoryLimit: 2
  source:
    repoURL: 'https://github.com/kubernetes-sigs/aws-ebs-csi-driver.git'
    path: charts/aws-ebs-csi-driver
    targetRevision: helm-chart-aws-ebs-csi-driver-2.27.0
    helm:
      valueFiles:
        - values.yaml
      parameters:
        - name: controller.serviceAccount.create
          value: 'false'
        - name: controller.serviceAccount.name
          value: ebs-csi-driver
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

[gxpd-jjh]

I am not deep enough here, but this smells weird:

aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/node.yaml

Line 14 in 33fffa3

{{- if .Values.a1CompatibilityDaemonSet }}

But its Value is commented out?

aws-ebs-csi-driver/charts/aws-ebs-csi-driver/values.yaml

Lines 413 to 423 in 33fffa3

	additionalDaemonSets:
	# Additional node DaemonSets, using the node config structure
	# See docs/additional-daemonsets.md for more information
	#
	# example:
	# nodeSelector:
	# node.kubernetes.io/instance-type: c5.large
	# volumeAttachLimit: 15
	# Enable compatibility for the A1 instance family via use of an AL2-based image in a separate DaemonSet
	# a1CompatibilityDaemonSet: true

[bryanfang]

Hi @bryanfang, Did you find a solution ? I've have the same issue.

Hi @barbaluc ,

I did not try to resolve the code issue with this repo, and have to modify volume size manually by following below steps:

1. kubectl patch sc ebs-sc -p '{"allowVolumeExpansion": true}'
2. kubectl get sc, you should see
   ALLOWVOLUMEEXPANSION is true
3. kubectl get pvc -n
4. kubectl edit pvc -n => change storage: 20Gi to 60Gi and save it
5. restart prometheus pod:
   kubectl delete pod -n

hope this helps!
BR,
Bryan

[barbaluc]

Hi @bryanfang , @gxpd-jjh,
I've upgraded my helm provider in terraform and it's working now.
Version 1.3.2 -> 2.12.1
I guess the helm version used was to old and not compatible with the syntax wrotte in the new template files.

[gxpd-jjh]

Hi @bryanfang , @gxpd-jjh, I've upgraded my helm provider in terraform and it's working now. Version 1.3.2 -> 2.12.1 I guess the helm version used was to old and not compatible with the syntax wrotte in the new template files.

Maybe I need to upgrade my argo instance; i'll give it a shot over next day or four.

[bryanfang]

@barbaluc @gxpd-jjh V2.12 has no issue, like my initial post, the issue comes up when I try to upgrade from v2.20 to v2.25, because in v2.25 the end driver is v1.25, this version of driver can update volume size automatically when I change the terraform value

[jcdauchy-moodys]

Went back to using 2.22.1 helm chart waiting for a new fixed HELM chart or explanation of how to use the values to fix this problem.

Thanks @gxpd-jjh

[barbaluc]

@bryanfang I think we don't talk about same components. I had the same issue like you when I've upgraded the csi driver chart version. And I've fixed it when I've upgraded the helm terraform provider version.

[jcdauchy-moodys]

I don't use terraform, I am using helm. I install the chart manually. Could it be my helm version.

The helm is not intended to be installed as a standalone chart ?

[bryanfang]

I don't use terraform, I am using helm. I install the chart manually. Could it be my helm version.

The helm is not intended to be installed as a standalone chart ?

After you change the volume size in chart, and once after deploy it, did you check if the volume size gets changed?

[bryanfang]

@bryanfang I think we don't talk about same components. I had the same issue like you when I've upgraded the csi driver chart version. And I've fixed it when I've upgraded the helm terraform provider version.

I think we are talking about the same issue, node parser issue with version v2.25.0

[gxpd-jjh]

Hi @bryanfang , @gxpd-jjh, I've upgraded my helm provider in terraform and it's working now. Version 1.3.2 -> 2.12.1 I guess the helm version used was to old and not compatible with the syntax wrotte in the new template files.

Maybe I need to upgrade my argo instance; i'll give it a shot over next day or four.

Upgrading my argo from an old version to latest did the trick. Something must have changed in underlying components.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[ConnorJC3]

/close

Appears to be an issue in third party components (argocd and/or terraform). Please reopen this issue or create a new issue if further support is needed.

[k8s-ci-robot]

@ConnorJC3: Closing this issue.

In response to this:

/close

Appears to be an issue in third party components (argocd and/or terraform). Please reopen this issue or create a new issue if further support is needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.