-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Unable to upgrade Kubernetes cluster from 1.25.x to 1.26.x due to deprecated API error #3984
Comments
Exact same issue here. |
@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to. |
you've possibly just saved my keyboard ... this 'feature' is such a PITA. By all means warn me about these things, but at least give me the option to opt out easily. |
This is a bug. The error occurs even when no HPA resources exist.
The APIService seems to be watching for the existence of this version and reporting it deprecated which is causing the upgrade to fail. |
You're absolutely right. I noticed that a few hours after replying to your post and forgot to come back and correct myself. Awkward x) |
Quick question. Do you have any hpa's at all. What I have noticed is that it also checks the last-applied-configuration annotation. If that annotation has reference to the old API it will fail. Looking forward to your reply. |
We encountered this bug without any hpa's at all:
|
Thanks for this!! This fixed it for us :D |
It also worked for us |
@sunu @tommy-dunn Please check the error message - it will tell the verb used. Please also check the Diagnose and solve problems page, to find out the user agent. A known issue is older KSM would use deprecated HPA api. kubernetes/kube-state-metrics#1711 Monitoring components such as datadog, may also bundle kube-state-metrics DataDog/datadog-agent#14487 In such cases, please update the installed KSM version accordingly to remove the usage, or use force upgrade to bypass the validation. |
you just saved my day. It works as described, thank you 🙏 |
Same problem here, I was stuck during the upgrade of my What I did@wenjungaogaogao I like your suggestion which was the right approach for me, I managed to upgrade my cluster to This wasn't so easy because After a lot of browsing on Github and Bitnami I finally found the right version of As you mentioned, this issue was fixed and embedded in release 2.7.0 Bitnami is a great library of tools and has a great community but it's quite complex to find a changelog for a chart and also identify which version can be used in order to target a specific version... Anyway, I did the reverse engineering by pulling helm charts, unpacking sub-charts, etc. to find target version for ℹ️ Here is a diagram for a better understanding : I reviewed each difference between
Then I deployed this new version in my cluster After a new attempt to upgrade the control plane, there still was the same error, which confused me a lot 😕 According to https://learn.microsoft.com/en-us/azure/aks/stop-cluster-upgrade-api-breaking-changes#before-you-begin you have to wait 12 hours from the last time the last deprecated API usage was seen 12h later, the control plane finally upgraded to version In conslusionIMHO bypassing this warning by forcing the upgrade as suggested is not the best solution, although it can save someone's day 😄 Hope this will help someone else |
It looks like this is actually multiple different issues that maybe should be split?
There was a great post by @aurel4oxand explaining how they managed to upgrade without forcing which is what I would ideally like to do. However, I am already on kube-state-metrics appversion 2.3.0 via prometheus community kube-state-metrics helm chart but this issue still occurs. I don't see the
The kube-state-metrics issue was good to know about but wouldn't there still have been a query we could use to see the I don't know how to check the actual apiVersion of each resource because of the kubectl behavior of returning whichever version it is queried with
|
Hi @emily-zall First of all, you're right : there are two topics in a single Github issue 👍 Regarding your problem, it seems you're using kube-state-metrics app Maybe I'm wrong but I think you should use this minimal version of kube-state-metrics Helm chart, which pulls kube-state-metrics app Best regards |
Thanks @aurel4oxand! So for people who want to find and resolve the deprecated API issue - it's not necessarily that there is a horizontalpod autoscaler it's that something is watching for one, as indicated by the kube-state-metrics is a common culprit and if so you will see it on the deprecated APIs page and there is great info above about how to update it. |
Describe the bug
I'm trying to upgrade my Kubernetes cluster from 1.25.x to 1.26.x but when I try to do this through the Azure portal or Azure CLI I get the following error message:
According to the documentation this is a preview feature, but I don't remember opting in to it and I can't find any reference to it when I run
az feature list
either.Now, to be clear, there is indeed an application in my cluster that is using deprecated APIs for some of its operations, but those are non-critical and fixing or removing the application as a whole is not a priority for me right now. I'd like to be have the ability to override checks like this depending on my own requirements. We already analyze the changelog of every Kubernetes upgrade and make plans for upgrades based on that, so this check is just getting in our way.
I followed the suggestion in the documentation to disable the check with the
az aks update
command, which seems to have done the trick, but it did take a very long time to run (almost 4 minutes), and that seems to be just a temporary fix.So my questions are as follows:
To Reproduce
Steps to reproduce the behavior:
Expected behavior
This preview feature can be disabled or bypassed with little effort.
Screenshots
N/A
Environment (please complete the following information):
Additional context
N/A
The text was updated successfully, but these errors were encountered: