-
Notifications
You must be signed in to change notification settings - Fork 80
Issue upgrading tomcat 0.3.0 to 0.4.x #162
Comments
@Nuru I could not reproduce the error by doing the following:
The tomcat chart creates a Deployment and Service. I agree that Can you please provide a sequence of commands that reproduce your error? (Alternatively, can you please explain why my sequence of commands would fail to reproduce the issue.) Thank you! |
I published a script to add the needed metadata. After running @dlipovetsky I have not made an exhaustive or definitive study of exactly what causes the error. My impression is that if you use helm3 to apply the same chart with the same values as you last used with helm2, helm3 will simply make the metadata changes. The error arises when you try to make substantive changes to existing resources in the release at the same time. You upgraded the
|
@dlipovetsky so far I have not reproduced this under Kubernetes 1.16. Reviewing my logs I can see that previously this occurred under Kubernetes 1.15 when updating the
I have confirmed the above fails on a cluster running Kubernetes 1.15.10. I have confirmed that if the helm2 install is of 0.4.0 the upgrade to 0.4.1 succeeds even on Kubernetes 1.15. Unfortunately, you cannot install I have yet to find a chart that has versions that can deploy on Kubernetes 1.16, one with Ingress apiVersion |
@Nuru Thanks for raising this issue. I will try my best to explain this by first providing some background on how it all fits together. The PR helm/helm#7649 was added in Helm 3.2.0 to provide functionality where orphaned kubernetes resources (not associated to a Helm release) in a cluster may under certain circumstances be associated to a Helm release. It is optional capability and there has been some open issues with it, such as helm/helm#8350. The If you Re #147, the OP had a number of different issues which I tried to explain 1 by 1 as they were independent of each other. His main problem seemed to be a CRD issue in the end as he explained in #147 (comment). Sometimes a symptom shown by a Helm error may be caused by a different issue and not related to your issue. Getting back to your issue. Why does a resource already exist for you when you are trying to upgrade/install? That is your issue and helm/helm#7649 is a potential solution if that is what you want to do. If it is then you need to add the Let me know if this helps answer your issue. |
@hickeyma wrote:
That is what I thought, but it did not happen that way, which is why I opened this issue.
Why would a resource not exist when I am trying to upgrade? The usual case for an upgrade is to modify existing resources. As I explained above, The key issue is that the upgrade of the converted release fails and it has nothing to do with CRDs. I have given you a reproducible case under Kubernetes 1.15. The general issue seems to be with resources that were present in the helm2 version and would be deleted by the new chart applied with helm3. Somehow the conversion process is not sufficient for helm3 to be comfortable deleting the resources because it is not sure it owns it. |
Rephrasing, please correct me if I'm wrong. A chart template implemented the label Because that label was created with the content of "Tiller" the 2to3 conversion keeps that label the same. Helm 3 requires that label to be "Helm" or else it fails the ownership check when attempting to upgrade to a newer chart version. |
@joejulian The label The label This brings us to the issue raised by the OP (@Nuru ). Looking at the error message again from the description above: Error: UPGRADE FAILED: rendered manifests contain a resource that already exists.
Unable to continue with update: Deployment "web-api" in namespace "dev" exists and cannot be imported into the current release:
invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm";
annotation validation error: missing key "meta.helm.sh/release-name": must be set to "web-api-dev";
annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "dev" The PR helm/helm#7649 functionality now kicks in as it provides the user with the option to associate this orphaned/unexplained resource to the release. For this to happen it is saying that the I hope this helps shed more light on the issue as it is tricky to explain all the parts as they are not all one atomic entity but different parts. |
I'm also thinking it may be unfixable. In the cert-manager chart linked above, they used that label as part of the selector. Any template that does this cannot be upgraded because the selector is immutable. |
@hickeyma wrote:
I explained the steps to reproduce this consistently in #162 (comment). If you do not have a Kubernetes 1.15 cluster, you can create one on AWS with EKS for probably only a few dollars if you just leave it running long enough to run the tests. You can also create a local Kubernetes cluster using Docker Desktop, and I believe Docker Desktop version 2.2.0.5 for Mac or Windows will install Kubernetes 1.15 for you. Note that the chart used to reproduce the problem, |
Thanks for responding @Nuru. I am able to reproduce it now and not sure why I was unable previously but maybe because I didn't update the version in the upgrade. My apologies. $ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-09-14T08:06:34Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
$ helm3 plugin list
NAME VERSION DESCRIPTION
2to3 0.6.0 migrate and cleanup Helm v2 configuration and releases in-place to Helm v3
$ helm2 install --name tomcat stable/tomcat --version 0.3.0
NAME: tomcat
LAST DEPLOYED: Thu Sep 24 11:51:10 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
tomcat-7dcf5886c9-2xxfx 0/1 Init:0/1 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tomcat LoadBalancer 10.111.21.76 <pending> 80:30244/TCP 0s
==> v1beta2/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
tomcat 0/1 1 0 0s
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
$ helm 2to3 convert tomcat
2020/09/24 11:53:46 Release "tomcat" will be converted from Helm v2 to Helm v3.
2020/09/24 11:53:46 [Helm 3] Release "tomcat" will be created.
2020/09/24 11:53:46 [Helm 3] ReleaseVersion "tomcat.v1" will be created.
2020/09/24 11:53:46 [Helm 3] ReleaseVersion "tomcat.v1" created.
2020/09/24 11:53:46 [Helm 3] Release "tomcat" created.
2020/09/24 11:53:46 Release "tomcat" was converted successfully from Helm v2 to Helm v3.
2020/09/24 11:53:46 Note: The v2 release information still remains and should be removed to avoid conflicts with the migrated v3 release.
2020/09/24 11:53:46 v2 release information should only be removed using `helm 2to3` cleanup and when all releases have been migrated over.
$ helm3 upgrade tomcat stable/tomcat --version 0.4.1
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Deployment "tomcat" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "tomcat"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default" |
I am unsure yet why the Helm 3 upgrade is checking the default pod/tomcat-569ff5bf4d-4rsfr 0/1 Pending 0 136m
default pod/tomcat-7dcf5886c9-2xxfx 1/1 Running 0 158m I am unsure if this is expected behavior or not. This might be triggering the adopt functionality under he hood. This is something that will need to be investigated further in the Helm engine and with the tomcat Helm chart. In the meantime, I recommend working around this issue by performing an upgrade which is identical to the previous deployment (install or upgrade). So in the case of $ helm2 install --name tomcat stable/tomcat --version 0.3.0
NAME: tomcat
LAST DEPLOYED: Thu Sep 24 11:51:10 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
tomcat-7dcf5886c9-2xxfx 0/1 Init:0/1 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tomcat LoadBalancer 10.111.21.76 <pending> 80:30244/TCP 0s
==> v1beta2/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
tomcat 0/1 1 0 0s
$ helm 2to3 convert tomcat
2020/09/24 11:53:46 Release "tomcat" will be converted from Helm v2 to Helm v3.
2020/09/24 11:53:46 [Helm 3] Release "tomcat" will be created.
2020/09/24 11:53:46 [Helm 3] ReleaseVersion "tomcat.v1" will be created.
2020/09/24 11:53:46 [Helm 3] ReleaseVersion "tomcat.v1" created.
2020/09/24 11:53:46 [Helm 3] Release "tomcat" created.
2020/09/24 11:53:46 Release "tomcat" was converted successfully from Helm v2 to Helm v3.
2020/09/24 11:53:46 Note: The v2 release information still remains and should be removed to avoid conflicts with the migrated v3 release.
2020/09/24 11:53:46 v2 release information should only be removed using `helm 2to3` cleanup and when all releases have been migrated over.
$ helm3 ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
tomcat default 1 2020-09-24 11:51:10.962369258 +0000 UTC deployed tomcat-0.3.0 7.0
$ helm3 upgrade tomcat stable/tomcat
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Deployment "tomcat" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "tomcat"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
$ helm3 upgrade tomcat stable/tomcat --version 0.3.0
Release "tomcat" has been upgraded. Happy Helming!
NAME: tomcat
LAST DEPLOYED: Thu Sep 24 12:12:35 2020
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
$ helm3 ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
tomcat default 2 2020-09-24 12:12:35.785146187 +0000 UTC deployed tomcat-0.3.0 7.0
$ helm3 upgrade tomcat stable/tomcat --version 0.4.1
Release "tomcat" has been upgraded. Happy Helming!
NAME: tomcat
LAST DEPLOYED: Thu Sep 24 12:13:01 2020
NAMESPACE: default
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP: By repeating the I am hesitant for the plugin to make changes to the manifest blob as explained in #162 (comment) and #162 (comment). For now, I will open a doc PR against the plugin README to document this in the "Troubleshooting section". Let me know what you think @Nuru. |
@hickeyma wrote:
This is a problem that people will likely run into before documentation will help. It is nothing specific to the As it stands, |
If a person does then they do the upgrade as described in #162 (comment). It will not block them.
I am yet to see a recurring pattern with other charts and in other scenarios. It is an edge case which will be investigated to see if it can be handled. The workaround is in place for the moment. |
The pattern I see is that the first update to a release after running |
I think that it is correct in that it is reason for causing the unexpected situation in the cluster (i.e. pod nopt been replaced) but I think it is Kubernetes itself which is not handling the |
Firstly, the apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: {{ template "tomcat.fullname" . }}
labels:
app: {{ template "tomcat.name" . }}
[...] and apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "tomcat.fullname" . }}
labels:
app: {{ template "tomcat.name" . }} have different On Kubernetes 1.15, when installing (using Helm 3 only) tomcat version (0.3.0) it deploys as expected with one pod. However, when upgrading to tomcat version (0.4.1) an additional pod is deployed as follows: $ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-09-14T08:06:34Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
$ helm3 install tomcat stable/tomcat --version 0.3.0
NAME: tomcat
LAST DEPLOYED: Fri Oct 9 13:53:30 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat-7dcf5886c9-mntpr 1/1 Running 0 100s
$ helm3 upgrade tomcat stable/tomcat --version 0.4.1
Release "tomcat" has been upgraded. Happy Helming!
NAME: tomcat
LAST DEPLOYED: Fri Oct 9 13:55:33 2020
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat-569ff5bf4d-9hgdm 0/1 Pending 0 7s
tomcat-7dcf5886c9-mntpr 1/1 Running 0 2m10s
$ kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default tomcat 1/1 1 1 13m
kube-system coredns 2/2 2 2 15d
kube-system tiller-deploy 1/1 1 1 15d
local-path-storage local-path-provisioner 1/1 1 1 15d 2 things seems strange to me with this:
$ kubectl get deployment tomcat -o yaml | more
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
meta.helm.sh/release-name: tomcat
meta.helm.sh/release-namespace: default
creationTimestamp: "2020-10-09T13:53:31Z"
[..] If I clean the previous deployment, install fresh with version $ helm3 upgrade tomcat stable/tomcat --version 0.4.1
Release "tomcat" has been upgraded. Happy Helming!
NAME: tomcat
LAST DEPLOYED: Fri Oct 9 18:03:09 2020
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default tomcat -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo http://$SERVICE_IP:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat-569ff5bf4d-qb6rr 1/1 Running 0 3h12m |
I think there will be an issue with any chart as follows:
Kubernetes will probably handle it in much the same manner as #162 (comment). That is why I believe your issue is reproducible only as follows:
|
My suggestion would be to protect users from trying to upgrade with updated APIs where the deprecated API is still supported because the way Kubernetes tries to handle it. |
@hickeyma Your test in #162 (comment) is flawed because you used an unsupported version of
I suggest re-running your test with kubectl v1.16.15. |
@Nuru The test needs to be run in a cluster with Kubernetes 1.15 as the deprecated API is not removed in that version. Hence, I can show the issue happening when you try to upgrade a release with deprecated API, with a chart containing the new or supported API. |
Updated the issue title to better describe the issue. Analysis of issue described in #162 (comment). The issue is not with the plugin but a side effect of the Kubernetes API and Helm chart. Closing the issue therefore. |
I agree with @hickeyma here as well, this plugin cannot cover all use cases. |
@rimusz I disagree that this is an edge case that should not be supported. At any given time, for any release of Kubernetes since at least 1.14, Kubernetes supports some deprecated APIs and some replacement APIs. This issue arises any time someone tries to simultaneously upgrade from helm 2 to helm 3 and from deprecated to newly supported APIs. These are likely to go together as part of generally preparing for an upgrade to a new version of Kubernetes. |
I agree with you @Nuru that it is not an edge case and it should try and be supported. It is however not something that the plugin can solve as such. It is a issue that occur when you upgrade a release with deprecated Kubernetes APIs with a new version of the I will re-open and open an issue in Helm (sorry forgot to do this). |
I do not really agreed on |
@Nuru I have raised helm/helm#9014 in Helm core. |
The problem will still occur if the next version of the chart upgrades the @hickeyma I disagree with you changing the title of this issue. I was using the The error message:
is from |
I changed the title to better describe the problem. At this moment we are only able to reproduce this issue using the
Yes the error message is from Helm. Yes you are seeing this error after a conversion. But the underlying problem when you "peel back the onion" is fundamentally how Kubernetes resources containing deprecated APIs are upgraded by Kubernetes when |
The fundamental issue may have to do with just upgrading from one tomcat version to another: helm/helm#9014 (comment). |
Closing as repository is archived. |
After using
helm 2to3 convert <release>
, the release is still not in a proper state forhelm
version 3 to use, because it is missing required metadata. If you re-apply the current chart and values,helm
will fix it for you, but if you immediately try to update the release with something new, it fails with an error like:2to3 should automatically add the correct metadata when doing the
convert
. See Release Note at helm/helm#7649 (comment)Note: This is the same starting place as #147, but that was closed when the OP found and fixed a different problem.
The text was updated successfully, but these errors were encountered: