Kubernetes API removals on 1.25/1.26 might impact your Operator. How to deal with it?

[camilamacedo86]

📢 Kubernetes API removals on 1.25/1.26 might impact your Operator

Some APIs versions are deprecated and will no longer be served on the Kubernetes version 1.25/1.26 and consequently on vendors like Openshift 4.12/4.13. For further information, check the guide.

⚠️ Be aware that:

• Cluster admins MUST have installed a version of your Operator that can work successfully on Kubernetes 1.25+ before upgrading their cluster from 1.24 to 1.25. Otherwise, they might face workflow issues
• Unfurtunilly, it is not possible to detect these cases as it was done for Kubernetes 1.22 to help you out. Resources using these APIs are not provided via the bundle manifests, so we cannot check them with SDK commands such as operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.25
• It is recommended you begin testing your Operator on Kubernetes 1.23+ to check if your versions are or not using these APIs. In this case, you need to perform a full regression test and check the Kubernetes metrics and the Operator logs. (For further info and examples, check the section below: How can I find the usage of the deprecated APIs )

NOTE: If you distributed solutions to work on Openshift, you might want to look at the topic, which has some specific guidance.

🙆 When the errors will appear

It will be unlikely to find problems by only installing the Operator bundle on the cluster. These APIs are commonly used at runtime (e.g. reconciliations made when the user creates CRs to provision the Operand).

Example:

An Operator reconciles a Kind/API to create a CronJob using batch/v1beta1 or raises an Event using events.k8s.io/v1beta1. Then, these operations will not work from Kubernetes 1.25+. The API versions will not be found on the cluster, and you will probably check out errors in Operator logs (api not found).

💁‍♀️ What does it mean for you?

• You must ensure that you no longer use these API versions. Otherwise, any operation of your project which requires these versions will fail on Kubernetes versions 1.25/1.26+.

• You should also check if your Operator versions are using these APIs or not. Then, you can adequately inform its users.

Via the CSV, we can use the spec minKubeVersion to define the minimal version and the annotation operatorhub.io/ui-metadata-max-k8s-version to define what are the max and min versions where our Operator can work successfully. This information will be provided to your users via the OperatorHub website.

Examples

• How the max and min Kubernetes versions appear in the OperatorHub website:

• Use case scenario:

If your Operator version uses the batch/v1beta1 and the Kubernetes API for CRDs v1, you ought to use the minKubeVersion spec to define it as 1.16 ( since the API v1 for CRDs was introduced in this version ). Therefore, you ought to use the annotation operatorhub.io/ui-metadata-max-k8s-version: 1.24 so that your users can be aware that this version will not work successfully on Kubernetes 1.25:

kind: ClusterServiceVersion
metadata:
  annotations:
       operatorhub.io/ui-metadata-max-k8s-version: 1.24
....
spec:
  minKubeVersion: 1.16.0

🙋 What are the APIs? How can I upgrade my Operator?

You can find all apis/versions removed on 1.25/1.26 on the guide guide as the required steps to upgrade them. However, the following is a summary:

APIs removed from k8s 1.25:

Group	Version	Resource	How to upgrade
batch	v1beta1	cronjobs	use v1 instead of v1beta1. v1 has been available since k8s v1.21. No notable changes
discovery.k8s.io	v1beta1	endpointslices	use v1 instead of v1beta1. v1 has been available since k8s v1.21. (See the guide to check the notable changes)
events.k8s.io	v1beta1	events	use v1 instead of v1beta1. v1 has been available since k8s v1.19. (See the guide to check the notable changes)
autoscaling	v2beta1	horizontalpodautoscalers	use v2 instead of v2beta1. v2 has been available since k8s v1.23. (No notable changes)
policy	v1beta1	poddisruptionbudgets	use v1 instead of v1beta1. v1 has been available since k8s v1.21. (See the guide to check the notable changes.)
policy	v1beta1	podsecuritypolicies	Migrated to 3rd-party admission webhooks.
node.k8s.io	v1beta1	runtimeclasses	use v1 instead of v1beta1. v1 has been available since k8s v1.20. (No notable changes)

APIs removed from k8s 1.26:

Group	Version	Resource	How to upgrade
autoscaling	v2beta2	horizontalpodautoscalers	use v2 instead of v2beta2. v2 has been available since k8s v1.23(No notable changes)
flowcontrol.apiserver.k8s.io	v1beta1	flowschemas	use v1beta2 instead of v1beta1. v1beta2 has been available since k8s v1.23(No notable changes)
flowcontrol.apiserver.k8s.io	v1beta1	prioritylevelconfigurations	use v1beta2 instead of v1beta1. v1beta2 has been available since  k8s v1.23(No notable changes)

🙋‍♀️ How can I find the usage of the deprecated APIs:

1. Inspect your Operator source code. Look for manifests and imports using the removed versions, e.g.:

$ find . \|  grep -r "k8s.io/api/batch/v1beta1" 

Binary file ./.git/index matches
./vendor/[k8s.io/client-go/kubernetes/scheme/register.go](http://k8s.io/client-go/kubernetes/scheme/register.go):	batchv1beta1 "[k8s.io/api/batch/v1beta1](http://k8s.io/api/batch/v1beta1)"
./vendor/k8s.io/client-go/kubernetes/typed/batch/v1beta1/batch_client.go:	v1beta1 "k8s.io/api/batch/v1beta1"
./vendor/k8s.io/client-go/kubernetes/typed/batch/v1beta1/cronjob.go:	v1beta1 "[k8s.io/api/batch/v1beta1](http://k8s.io/api/batch/v1beta1)"
./vendor/k8s.io/api/batch/v1beta1/generated.pb.go:// source: k8s.io/kubernetes/vendor/k8s.io/api/batch/v1beta1/generated.proto
./vendor/k8s.io/api/batch/v1beta1/generated.pb.go:	proto.RegisterFile("k8s.io/kubernetes/vendor/k8s.io/api/batch/v1beta1/generated.proto", fileDescriptor_e57b277b05179ae7)
./vendor/k8s.io/api/batch/v1beta1/doc.go:package v1beta1 // import "[k8s.io/api/batch/v1beta1](http://k8s.io/api/batch/v1beta1)"
./vendor/modules.txt:k8s.io/api/batch/v1beta1
./pkg/controller/keycloakbackup/keycloakbackup_reconciler_test.go:	"k8s.io/api/batch/v1beta1"
./pkg/controller/keycloakbackup/keycloakbackup_controller.go:	"[k8s.io/api/batch/v1beta1](http://k8s.io/api/batch/v1beta1)"
./pkg/common/backup_state.go:	"k8s.io/api/batch/v1beta1"
./pkg/model/postgresql_aws_periodic_backup.go:	"[k8s.io/api/batch/v1beta1](http://k8s.io/api/batch/v1beta1)"

2. By running your Operator on the cluster, you can check that the Kubernetes API will also try to warn you when a workflow that uses these API versions is executed, e.g.:

1.651068606661642e+09	INFO	KubeAPIWarningLogger	batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
1.651068606756052e+09	INFO	controller.memcached	Creating a new CronJob	{"reconciler group": "[cache.example.com](http://cache.example.com/)", "reconciler kind": "Memcached", "name": "memcached-sample", "namespace": "default", "Cron.Namespace": "default", "Cron.Name": "memcached-sample"}

⚠️ The above test was made using Kind 0.12.0 and k8s 1.23+. Please, do the tests against Kubernetes API 1.23+ to ensure that the warnings will be fired.

3. Use the metrics to see if your Operator workflows are calling these deprecated APIs (which will be removed)

You can use Kubernetes metrics to check if your Operator calls the deprecated APIs (More info).

💡 Note that it is not easy to identify if your Operator is doing the calls. Other components running on the cluster can also have called them. In this way, we recommend that you; initialise a new cluster for these tests. Then, before beginning to test your Operator verify the results so that you can compare the calls to these APIs before your Operator and workflows are executed with its result at the end.

Example

• Before running the Operator and applying the CR, which has a code implementation calling the CronJob API v1beta1 that will be removed on k8s 1.25:

$ kubectl get --raw /metrics | prom2json | jq '
>   # set $deprecated to a list of deprecated APIs
>   [
>     .[] | 
>     select(.name=="apiserver_requested_deprecated_apis").metrics[].labels |
>     {group,version,resource}
>   ] as $deprecated 
>   
>   |
>   
>   # select apiserver_request_total metrics which are deprecated
>   .[] | select(.name=="apiserver_request_total").metrics[] |
>   select(.labels | {group,version,resource} as $key | $deprecated | index($key))
> '

Then, we checked as a result:

{
  "labels": {
    "code": "200",
    "component": "apiserver",
    "dry_run": "",
    "group": "policy",
    "resource": "podsecuritypolicies",
    "scope": "cluster",
    "subresource": "",
    "verb": "LIST",
    "version": "v1beta1"
  },
  "value": "1"
}

• Now, see the results after running the Operator and applying the CR. We can check that the Operator reconciliation called the API CronJob v1beta1 to LIST and POST:

{
  "labels": {
    "code": "200",
    "component": "apiserver",
    "dry_run": "",
    "group": "batch",
    "resource": "cronjobs",
    "scope": "cluster",
    "subresource": "",
    "verb": "LIST",
    "version": "v1beta1"
  },
  "value": "1"
}
{
  "labels": {
    "code": "200",
    "component": "apiserver",
    "dry_run": "",
    "group": "policy",
    "resource": "podsecuritypolicies",
    "scope": "cluster",
    "subresource": "",
    "verb": "LIST",
    "version": "v1beta1"
  },
  "value": "1"
}
{
  "labels": {
    "code": "201",
    "component": "apiserver",
    "dry_run": "",
    "group": "batch",
    "resource": "cronjobs",
    "scope": "resource",
    "subresource": "",
    "verb": "POST",
    "version": "v1beta1"
  },
  "value": "1"
}