Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add option to disable the use of --force flag when running helm upgrade extensions #4864

Closed
emosbaugh opened this issue Aug 15, 2024 · 0 comments · Fixed by #4866
Closed

Add option to disable the use of --force flag when running helm upgrade extensions #4864

emosbaugh opened this issue Aug 15, 2024 · 0 comments · Fixed by #4866
Labels
enhancement New feature or request

Comments

@emosbaugh
Copy link
Contributor

emosbaugh commented Aug 15, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

A chart with a PVC is unlikely to work when the --force option is used because the client side apply fails to overwrite dynamic values from the server.

Relevant helm issue:

helm/helm#11729 (comment)

Repro steps

install chart

helm repo add twuni https://helm.twun.io
helm pull twuni/docker-registry --version 2.2.3

root@ethanm-ec-1:/home/ethan# k0s kubectl apply -f k0schart.yaml
chart.helm.k0sproject.io/k0s-addon-chart-docker-registry created
root@ethanm-ec-1:/home/ethan# helm history -n registry docker-registry
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
REVISION	UPDATED                 	STATUS  	CHART                	APP VERSION	DESCRIPTION
1       	Thu Aug 15 20:23:26 2024	deployed	docker-registry-2.2.3	2.8.1      	Install complete

upgrade chart

root@ethanm-ec-1:/home/ethan# k0s kubectl apply -f upk0schart.yaml
chart.helm.k0sproject.io/k0s-addon-chart-docker-registry configured
root@ethanm-ec-1:/home/ethan# helm history -n registry docker-registry
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
REVISION	UPDATED                 	STATUS    	CHART                	APP VERSION	DESCRIPTION
1       	Thu Aug 15 20:23:26 2024	deployed  	docker-registry-2.2.3	2.8.1      	Install complete
2       	Thu Aug 15 20:27:39 2024	superseded	docker-registry-2.2.3	2.8.1      	Upgrade "docker-registry" failed: failed to replace object: PersistentVolumeClaim "docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound clai...
3       	Thu Aug 15 20:27:40 2024	failed    	docker-registry-2.2.3	2.8.1      	Rollback "docker-registry" failed: failed to replace object: PersistentVolumeClaim "docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound cla...

See the error in the logs

Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="E0815 20:27:40.205184 1261898 pod_workers.go:1298] \"Error syncing pod, skipping\" err=\"unmounted volumes=[data kube-api-access-q58bk], unattached volumes=[], failed to process volumes=[]: context canceled\" pod=\"registry/docker-registry-8485d69f49-r8gzj\" podUID=\"485fd833-85be-47e0-a47f-7b9ea7f46a2a\"" component=kubelet stream=stderr
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler" Chart="{k0s-addon-chart-docker-registry kube-system}" component=extensions_controller controller=chart controllerGroup=helm.k0sproject.io controllerKind=Chart name=k0s-addon-chart-docker-registry namespace=kube-system reconcileID="\"43d283ce-58e3-4356-9bd4-fab9c5301bd6\""
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=error msg="Reconciler error" Chart="{k0s-addon-chart-docker-registry kube-system}" component=extensions_controller controller=chart controllerGroup=helm.k0sproject.io controllerKind=Chart error="can't update or install chart: can't reconcile upgrade for \"k0s-addon-chart-docker-registry\": can't upgrade loadedChart `docker-registry`: an error occurred while rolling back the release. original upgrade error: failed to replace object: PersistentVolumeClaim \"docker-registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes:      {\"ReadWriteOnce\"},\n  \tSelector:         nil,\n  \tResources:        {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName:       \"pvc-3e8cf506-0a8b-438e-9fb1-4bd7b371d0ae\",\n+ \tVolumeName:       \"\",\n  \tStorageClassName: &\"local-path\",\n  \tVolumeMode:       &\"Filesystem\",\n  \t... // 3 identical fields\n  }\n: failed to replace object: PersistentVolumeClaim \"docker-registry\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes:      {\"ReadWriteOnce\"},\n  \tSelector:         nil,\n  \tResources:        {Requests: {s\"storage\": {i: {...}, s: \"10Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName:       \"pvc-3e8cf506-0a8b-438e-9fb1-4bd7b371d0ae\",\n+ \tVolumeName:       \"\",\n  \tStorageClassName: &\"local-path\",\n  \tVolumeMode:       &\"Filesystem\",\n  \t... // 3 identical fields\n  }\n" name=k0s-addon-chart-docker-registry namespace=kube-system reconcileID="\"43d283ce-58e3-4356-9bd4-fab9c5301bd6\""
Aug 15 20:27:40 ethanm-ec-1 k0s[1261741]: time="2024-08-15 20:27:40" level=info msg="I0815 20:27:40.213714 1261822 replica_set.go:676] \"Finished syncing\" logger=\"replicaset-controller\" kind=\"ReplicaSet\" key=\"registry/docker-registry-8485d69f49\" duration=\"15.477802ms\"" component=kube-controller-manager stream=stderr

root@ethanm-ec-1:/home/ethan# cat k0schart.yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: k0s-addon-chart-docker-registry
  namespace: kube-system
spec:
  chartName: /home/ethan/docker-registry-2.2.3.tgz
  namespace: registry
  releaseName: docker-registry
  timeout: 0s
  values: |2
    image:
      tag: 2.8.3
    persistence:
      accessMode: ReadWriteOnce
      enabled: true
      size: 10Gi
      storageClass: local-path
    replicaCount: 1
    storage: filesystem
  version: 2.2.3

root@ethanm-ec-1:/home/ethan# cat upk0schart.yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: k0s-addon-chart-docker-registry
  namespace: kube-system
spec:
  chartName: /home/ethan/docker-registry-2.2.3.tgz
  namespace: registry
  releaseName: docker-registry
  timeout: 0s
  values: |2
    image:
      tag: 2.8.3
    persistence:
      accessMode: ReadWriteOnce
      enabled: true
      size: 10Gi
      storageClass: local-path
    replicaCount: 1
    storage: filesystem
    extraEnvVars: []
      - name: TESTING
        value: "testing"
  version: 2.2.3

Helm CLI repro

ethan@Ethans-MacBook-Pro-2 scratch % helm install registry twuni/docker-registry --version 2.2.3 --namespace registry --create-namespace --values values.yaml
NAME: registry
LAST DEPLOYED: Thu Aug 15 13:21:40 2024
NAMESPACE: registry
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace registry -l "app=docker-registry,release=registry" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl -n registry port-forward $POD_NAME 8080:5000

helm upgrade --install --force registry twuni/docker-registry --version 2.2.3 --namespace registry --values upvalues.yaml --debug
history.go:56: [debug] getting history for release registry
upgrade.go:150: [debug] preparing upgrade for registry
upgrade.go:158: [debug] performing update for registry
upgrade.go:330: [debug] creating upgraded release for registry
client.go:385: [debug] checking 5 resources for changes
client.go:670: [debug] Replaced "registry-docker-registry-secret" with kind Secret for kind Secret
client.go:670: [debug] Replaced "registry-docker-registry-config" with kind ConfigMap for kind ConfigMap
client.go:417: [debug] error updating the resource "registry-docker-registry":
	 failed to replace object: PersistentVolumeClaim "registry-docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "2Gi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-efd7b0c0-6195-488d-8cce-dfae3ab2f9ca",
+ 	VolumeName:       "",
- 	StorageClassName: &"local-path",
+ 	StorageClassName: nil,
  	VolumeMode:       &"Filesystem",
  	DataSource:       nil,
  	... // 2 identical fields
  }

client.go:670: [debug] Replaced "registry-docker-registry" with kind Service for kind Service
client.go:670: [debug] Replaced "registry-docker-registry" with kind Deployment for kind Deployment
upgrade.go:442: [debug] warning: Upgrade "registry" failed: failed to replace object: PersistentVolumeClaim "registry-docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "2Gi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-efd7b0c0-6195-488d-8cce-dfae3ab2f9ca",
+ 	VolumeName:       "",
- 	StorageClassName: &"local-path",
+ 	StorageClassName: nil,
  	VolumeMode:       &"Filesystem",
  	DataSource:       nil,
  	... // 2 identical fields
  }

Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "registry-docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "2Gi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-efd7b0c0-6195-488d-8cce-dfae3ab2f9ca",
+ 	VolumeName:       "",
- 	StorageClassName: &"local-path",
+ 	StorageClassName: nil,
  	VolumeMode:       &"Filesystem",
  	DataSource:       nil,
  	... // 2 identical fields
  }

helm.go:84: [debug] failed to replace object: PersistentVolumeClaim "registry-docker-registry" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "2Gi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-efd7b0c0-6195-488d-8cce-dfae3ab2f9ca",
+ 	VolumeName:       "",
- 	StorageClassName: &"local-path",
+ 	StorageClassName: nil,
  	VolumeMode:       &"Filesystem",
  	DataSource:       nil,
  	... // 2 identical fields
  }

helm.sh/helm/v3/pkg/kube.(*Client).Update
	helm.sh/helm/v3/pkg/kube/client.go:430
helm.sh/helm/v3/pkg/action.(*Upgrade).releasingUpgrade
	helm.sh/helm/v3/pkg/action/upgrade.go:384
runtime.goexit
	runtime/asm_arm64.s:1172
UPGRADE FAILED
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:209
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.6.1/command.go:968
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_arm64.s:1172

ethan@Ethans-MacBook-Pro-2 scratch % cat values.yaml
image:
  tag: 2.8.3
persistence:
  accessMode: ReadWriteOnce
  enabled: true
  size: 2Gi
replicaCount: 1
storage: filesystem

ethan@Ethans-MacBook-Pro-2 scratch % cat upvalues.yaml
image:
  tag: 2.8.3
persistence:
  accessMode: ReadWriteOnce
  enabled: true
  size: 2Gi
replicaCount: 1
storage: filesystem
extraEnvVars: []
  - name: TESTING
    value: "testing"

Describe the solution you would like

Add DisableForce boolean to the Chart object in the k0s ClusterSpec as well as the ChartSpec object in the chart CR.

Describe alternatives you've considered

Fork the twuni/docker-registry chart and use a StatefulSet or use helm's lookup() function to set the VolumeName and StorageClassName.

Additional context

No response
@emosbaugh emosbaugh added the enhancement New feature or request label Aug 15, 2024
@emosbaugh emosbaugh changed the title The use of --force when applying helm upgrades causes issues with some charts Add DisableForce bool to ClusterConfig Chart object as well as Chart CR to disable the use of --force flag when running helm upgrade Aug 15, 2024
@emosbaugh emosbaugh changed the title Add DisableForce bool to ClusterConfig Chart object as well as Chart CR to disable the use of --force flag when running helm upgrade Add option to disable the use of --force flag when running helm upgrade extensions Aug 15, 2024
@emosbaugh emosbaugh mentioned this issue Aug 16, 2024
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(helm): add option to disable helm upgrade force flag emosbaugh/k0s
1 participant
@emosbaugh