[BUG] Unable to upgrade Kubernetes cluster from 1.25.x to 1.26.x due to deprecated API error

[OvervCW]

Describe the bug
I'm trying to upgrade my Kubernetes cluster from 1.25.x to 1.26.x but when I try to do this through the Azure portal or Azure CLI I get the following error message:

(ValidationError) Control Plane upgrade is blocked due to recent usage of a Kubernetes API deprecated in the specified version. Please refer to https://kubernetes.io/docs/reference/using-api/deprecation-guide to migrate the usage. To bypass this error, set forceUpgrade in upgradeSettings.overrideSettings. Bypassing this error without migrating usage will result in the deprecated Kubernetes API calls failing. See details in https://aka.ms/aks/UpgradeAndDeprecatedAPIs. Usage details: 1 error occurred:
	* usage has been detected on API autoscaling.horizontalpodautoscalers.v2beta2.watch, and was recently seen at: 2023-11-07 12:53:43 +0000 UTC, which will be removed in 1.26

According to the documentation this is a preview feature, but I don't remember opting in to it and I can't find any reference to it when I run az feature list either.

Now, to be clear, there is indeed an application in my cluster that is using deprecated APIs for some of its operations, but those are non-critical and fixing or removing the application as a whole is not a priority for me right now. I'd like to be have the ability to override checks like this depending on my own requirements. We already analyze the changelog of every Kubernetes upgrade and make plans for upgrades based on that, so this check is just getting in our way.

I followed the suggestion in the documentation to disable the check with the az aks update command, which seems to have done the trick, but it did take a very long time to run (almost 4 minutes), and that seems to be just a temporary fix.

So my questions are as follows:

• Why does this check appear to be enabled for all Kubernetes clusters when it's a preview feature?
• Why is it not possible to override the check from the Azure portal interface?
• Is it possible to opt out of this check altogether?

To Reproduce
Steps to reproduce the behavior:

1. Create a Kubernetes 1.25.x cluster.
2. Run an application on it that uses deprecated APIs.
3. Try to upgrade the cluster to 1.26.x.

Expected behavior
This preview feature can be disabled or bypassed with little effort.

Screenshots
N/A

Environment (please complete the following information):

• Azure CLI 2.53.1
• Azure CLI aks-preview extension 0.5.170
• Kubernetes 1.25.5

Additional context
N/A

[williambonomo]

Exact same issue here.
Using --enable-force-upgrade and --upgrade-override-until as per Microsoft articles doesn't work. Such a joke.
Got this error even using aks CLI preview: unrecognized arguments: --enable-force-upgrade --upgrade-override-until 2023-11-10T13:00:00Z

[OvervCW]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

[reschex]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

you've possibly just saved my keyboard ... this 'feature' is such a PITA. By all means warn me about these things, but at least give me the option to opt out easily.

[prov-tommy-dunn]

This is a bug. The error occurs even when no HPA resources exist.

kubectl get hpa.v2beta2.autoscaling -o wide --all-namespaces
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler

No resources found 

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

# HELP apiserver_requested_deprecated_apis [STABLE] Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release.
# TYPE apiserver_requested_deprecated_apis gauge
apiserver_requested_deprecated_apis{group="autoscaling",removed_release="1.26",resource="horizontalpodautoscalers",subresource="",version="v2beta2"} 1

% kubectl api-resources --no-headers  |while read type ignore; do kubectl get $type  -A -o go-template='{{range $index,$pod := .items}}{{(or .metadata.name  "-")}} {{(or .metadata.namespace "-")}} {{ (or .kind "-")}} {{ (or .apiVersion  "-") }} {{"\n"}}{{end}}'  2>/dev/null; done | grep v2beta2

v2beta2.autoscaling - APIService apiregistration.k8s.io/v1 

The APIService seems to be watching for the existence of this version and reporting it deprecated which is causing the upgrade to fail.

[williambonomo]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

You're absolutely right. I noticed that a few hours after replying to your post and forgot to come back and correct myself. Awkward x)

[PixelRobots]

This is a bug. The error occurs even when no HPA resources exist.

kubectl get hpa.v2beta2.autoscaling -o wide --all-namespaces
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler

No resources found 

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

# HELP apiserver_requested_deprecated_apis [STABLE] Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release.
# TYPE apiserver_requested_deprecated_apis gauge
apiserver_requested_deprecated_apis{group="autoscaling",removed_release="1.26",resource="horizontalpodautoscalers",subresource="",version="v2beta2"} 1

% kubectl api-resources --no-headers  |while read type ignore; do kubectl get $type  -A -o go-template='{{range $index,$pod := .items}}{{(or .metadata.name  "-")}} {{(or .metadata.namespace "-")}} {{ (or .kind "-")}} {{ (or .apiVersion  "-") }} {{"\n"}}{{end}}'  2>/dev/null; done | grep v2beta2

v2beta2.autoscaling - APIService apiregistration.k8s.io/v1 

The APIService seems to be watching for the existence of this version and reporting it deprecated which is causing the upgrade to fail.

Quick question. Do you have any hpa's at all.

What I have noticed is that it also checks the last-applied-configuration annotation. If that annotation has reference to the old API it will fail.

Looking forward to your reply.

[sunu]

Quick question. Do you have any hpa's at all.

We encountered this bug without any hpa's at all:

$ kubectl get hpa --all-namespaces
No resources found

[HanfiroBoy]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

Thanks for this!! This fixed it for us :D

[michasacuer]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

It also worked for us

[wenjungaogaogao]

@sunu @tommy-dunn

Please check the error message - it will tell the verb used.

Please also check the Diagnose and solve problems page, to find out the user agent.

A known issue is older KSM would use deprecated HPA api. kubernetes/kube-state-metrics#1711

Monitoring components such as datadog, may also bundle kube-state-metrics DataDog/datadog-agent#14487

In such cases, please update the installed KSM version accordingly to remove the usage, or use force upgrade to bypass the validation.

[istvanfazakas]

@williambonomo I was stuck on that exact same problem as well due to the confusing documentation, but I later found out that you have to use those arguments with the az aks update command instead of az aks upgrade. You basically have to first use that command to enable the forced upgrades and then use the actual upgrade command that you're used to.

you just saved my day. It works as described, thank you 🙏

[aurel4oxand]

@sunu @tommy-dunn

Please check the error message - it will tell the verb used.

Please also check the Diagnose and solve problems page, to find out the user agent.

A known issue is older KSM would use deprecated HPA api. kubernetes/kube-state-metrics#1711

Monitoring components such as datadog, may also bundle kube-state-metrics DataDog/datadog-agent#14487

In such cases, please update the installed KSM version accordingly to remove the usage, or use force upgrade to bypass the validation.

Same problem here, I was stuck during the upgrade of my 1.25.x control plane, due to presence of deprecated API use (HPA), but I don't use HPA at all 😕

What I did

@wenjungaogaogao I like your suggestion which was the right approach for me, I managed to upgrade my cluster to v1.26.10 by upgrading kube-state-metrics Bitnami chart version, and then the problem was gone 🎉

This wasn't so easy because kube-state-metrics was pulled as a dependency by kube-prometheus Helm chart, I needed to upgrade this parent Helm chart first

After a lot of browsing on Github and Bitnami I finally found the right version of kube-prometheus with a kube-state-metrics dependency which fixed the deprecation issue

As you mentioned, this issue was fixed and embedded in release 2.7.0

Bitnami is a great library of tools and has a great community but it's quite complex to find a changelog for a chart and also identify which version can be used in order to target a specific version...

Anyway, I did the reverse engineering by pulling helm charts, unpacking sub-charts, etc. to find target version for kube-state-metrics, which led me to this :

ℹ️ kube-prometheus chart v8.3.1 is the minimal version to pull kube-state-metrics in v2.7.0 (which is packaged in Bitnami chart v3.2.6)

Here is a diagram for a better understanding :

I reviewed each difference between kube-prometheus versions v8.0.7 and v8.3.1 by using Bitnami's comparison tool to ensure that we won't break something in our Prometheus monitoring stack :

• Templates : https://artifacthub.io/packages/helm/bitnami/kube-prometheus/8.3.1?modal=template&template=alertmanager/alertmanager.yaml&compare-to=8.0.7
• Values : https://artifacthub.io/packages/helm/bitnami/kube-prometheus/8.3.1?modal=values&compare-to=8.0.7

Then I deployed this new version in my cluster

After a new attempt to upgrade the control plane, there still was the same error, which confused me a lot 😕

According to https://learn.microsoft.com/en-us/azure/aks/stop-cluster-upgrade-api-breaking-changes#before-you-begin you have to wait 12 hours from the last time the last deprecated API usage was seen

12h later, the control plane finally upgraded to version 1.26 🎉

In conslusion

IMHO bypassing this warning by forcing the upgrade as suggested is not the best solution, although it can save someone's day 😄

Hope this will help someone else

[emily-zall]

It looks like this is actually multiple different issues that maybe should be split?

1. The original issue says that they are using deprecated APIs and want an easy way to bypass that check. https://learn.microsoft.com/en-us/azure/aks/stop-cluster-upgrade-api-breaking-changes#bypass-validation-to-ignore-api-changes I see a Force Upgrade box in the UI but when I check the box I still get the same error telling me "to bypass this error, set forceUpgrade in upgradeSettings.overrideSettings" and had to also do az aks update -n devtest-us1 -g devtest-us1-rg --enable-force-upgrade

2. A deprecation warning on autoscaling/v2beta2 HorizontalPodAutoscaler will happen even if it's not actually in use and there are may be no HorizontalPodAutoscalers.

There was a great post by @aurel4oxand explaining how they managed to upgrade without forcing which is what I would ideally like to do. However, I am already on kube-state-metrics appversion 2.3.0 via prometheus community kube-state-metrics helm chart but this issue still occurs.

I don't see the autoscaling/v2beta2 HorizontalPodAutoscaler anywhere and it has been this way for more than 12 hours

➜ kubectl api-resources | grep horizontal
horizontalpodautoscalers                   hpa                      autoscaling/v2                                         true         HorizontalPodAutoscaler

The kube-state-metrics issue was good to know about but wouldn't there still have been a query we could use to see the autoscaling/v2beta2 HorizontalPodAutoscaler?

I don't know how to check the actual apiVersion of each resource because of the kubectl behavior of returning whichever version it is queried with

➜ kubectl get hpa -A -o custom-columns=Name:.metadata.name,API:.apiVersion
Name                    API
datadog-cluster-agent   autoscaling/v2
kube-state-metrics      autoscaling/v2
osm-controller-hpa      autoscaling/v2
osm-injector-hpa        autoscaling/v2

➜ kubectl get horizontalpodautoscalers.v2beta2.autoscaling -A
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
NAMESPACE            NAME                    REFERENCE                          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
datadog-operator     datadog-cluster-agent   Deployment/datadog-cluster-agent   70%/80%           2         12        1          229d
kube-state-metrics   kube-state-metrics      Deployment/kube-state-metrics      0%/75%            1         10        1          238d
kube-system          osm-controller-hpa      Deployment/osm-controller          28%/80%, 0%/80%   2         5         2          238d

[aurel4oxand]

It looks like this is actually multiple different issues that maybe should be split?

1. The original issue says that they are using deprecated APIs and want an easy way to bypass that check. https://learn.microsoft.com/en-us/azure/aks/stop-cluster-upgrade-api-breaking-changes#bypass-validation-to-ignore-api-changes I see a Force Upgrade box in the UI but when I check the box I still get the same error telling me "to bypass this error, set forceUpgrade in upgradeSettings.overrideSettings" and had to also do az aks update -n devtest-us1 -g devtest-us1-rg --enable-force-upgrade
2. A deprecation warning on autoscaling/v2beta2 HorizontalPodAutoscaler will happen even if it's not actually in use and there are may be no HorizontalPodAutoscalers.

There was a great post by @aurel4oxand explaining how they managed to upgrade without forcing which is what I would ideally like to do. However, I am already on kube-state-metrics appversion 2.3.0 via prometheus community kube-state-metrics helm chart but this issue still occurs.

I don't see the autoscaling/v2beta2 HorizontalPodAutoscaler anywhere and it has been this way for more than 12 hours

➜ kubectl api-resources | grep horizontal
horizontalpodautoscalers                   hpa                      autoscaling/v2                                         true         HorizontalPodAutoscaler

The kube-state-metrics issue was good to know about but wouldn't there still have been a query we could use to see the autoscaling/v2beta2 HorizontalPodAutoscaler?

I don't know how to check the actual apiVersion of each resource because of the kubectl behavior of returning whichever version it is queried with

➜ kubectl get hpa -A -o custom-columns=Name:.metadata.name,API:.apiVersion
Name                    API
datadog-cluster-agent   autoscaling/v2
kube-state-metrics      autoscaling/v2
osm-controller-hpa      autoscaling/v2
osm-injector-hpa        autoscaling/v2

➜ kubectl get horizontalpodautoscalers.v2beta2.autoscaling -A
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
NAMESPACE            NAME                    REFERENCE                          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
datadog-operator     datadog-cluster-agent   Deployment/datadog-cluster-agent   70%/80%           2         12        1          229d
kube-state-metrics   kube-state-metrics      Deployment/kube-state-metrics      0%/75%            1         10        1          238d
kube-system          osm-controller-hpa      Deployment/osm-controller          28%/80%, 0%/80%   2         5         2          238d

Hi @emily-zall

First of all, you're right : there are two topics in a single Github issue 👍

Regarding your problem, it seems you're using kube-state-metrics app v2.3.0 but the fix I was talking about was released in v2.7.0 (see my diagram above)

Maybe I'm wrong but I think you should use this minimal version of kube-state-metrics Helm chart, which pulls kube-state-metrics app v2.7.0

Best regards

[emily-zall]

Thanks @aurel4oxand!

So for people who want to find and resolve the deprecated API issue - it's not necessarily that there is a horizontalpod autoscaler it's that something is watching for one, as indicated by the .watch part of the error message. The Azure Deprecated API Resources page will actually show you which resource specifically it is if you look through that whole chart. I haven't found any way to get this specific information via CLI so far. More details here: https://stackoverflow.com/a/78125579/9587902

kube-state-metrics is a common culprit and if so you will see it on the deprecated APIs page and there is great info above about how to update it.