[Bug]: Azure Provider Containerservice does not work after update >= 1.2.0

[Crazyigor1987]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

containerservice.azure.upbound.io/v1beta1

Resource MRs required to reproduce the bug

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: debug-config
spec:
  args:
    - --debug
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-storage
spec:
  package: xpkg.upbound.io/upbound/provider-azure-storage:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-network
spec:
  package: xpkg.upbound.io/upbound/provider-azure-network:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-keyvault
spec:
  package: xpkg.upbound.io/upbound/provider-azure-keyvault:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-managedidentity
spec:
  package: xpkg.upbound.io/upbound/provider-azure-managedidentity:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-compute
spec:
  package: xpkg.upbound.io/upbound/provider-azure-compute:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-containerservice
spec:
  package: xpkg.upbound.io/upbound/provider-azure-containerservice:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-authorization
spec:
  package: xpkg.upbound.io/upbound/provider-azure-authorization:v1.3.0
  controllerConfigRef:
    name: debug-config

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-family-azure
spec:
  package: xpkg.upbound.io/upbound/provider-family-azure:v1.3.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-provider-argocd
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-argocd:v0.7.0
  controllerConfigRef:
    name: debug-config
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.14.0
  controllerConfigRef:
    name: debug-config

Steps to Reproduce

I have updated the provider configs from version 1.1.0 to 1.3.0.

What happened?

After the update, i am not able to get informations about the running clusters created by the containerservice api.

user@local:~$ kubectl get kubernetesclusters.containerservice.azure.upbound.io 
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

Pods and crossplane are running

user@local:~$ kubectl -n crossplane get pods
NAME                                                             READY   STATUS    RESTARTS       AGE
crossplane-785f969dc5-62xmf                                      2/2     Running   0              5d20h
crossplane-785f969dc5-j4bn7                                      2/2     Running   0              5d23h
crossplane-785f969dc5-w6kdz                                      2/2     Running   0              5d23h
crossplane-rbac-manager-84b7bf46db-r6lx8                         2/2     Running   10 (15h ago)   5d23h
provider-azure-authorization-ca718ab078b2-5fc767b8bc-x92mp       2/2     Running   0              6m53s
provider-azure-compute-5f7e12c15a84-776488c659-92lgk             2/2     Running   0              6m45s
provider-azure-containerservice-7521bcd1e3c1-589f7b58f9-rlq62    2/2     Running   0              6m59s
provider-azure-keyvault-5e7163c6af8f-7c4dc77644-z2dvp            2/2     Running   0              7m1s
provider-azure-managedidentity-ef0465e36bb5-646d6767f6-vsqtd     2/2     Running   0              6m47s
provider-azure-network-8ade55f8f04c-657dd6cb67-fprg5             2/2     Running   0              6m43s
provider-azure-storage-a1be2f8ec957-6fff94df49-5d77l             2/2     Running   0              6m51s
provider-kubernetes-f8518c887488-76f4dcb5f4-87c58                2/2     Running   0              5d23h
provider-terraform-ff16e1bc51b9-7d54565d4c-pq529                 2/2     Running   0              5d17h
upbound-provider-family-azure-b3f8f01fe9bd-85f7469fcd-hs8bv      2/2     Running   0              7m2s
upbound-provider-provider-argocd-899b2f59c6aa-7f9585459f-bvfcc   2/2     Running   0              5d21h

Relevant Error Output Snippet

The logs give no indication of misbehaviour.

user@local:~$ kubectl -n crossplane logs provider-azure-containerservice-7521bcd1e3c1-589f7b58f9-rlq62 package-runtime
2024-06-18T11:20:10Z	DEBUG	provider-azure	Starting	{"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}

Crossplane Version

crossplane/crossplane:v1.16.0

Provider Version

1.3.0

Kubernetes Version

v1.27.9

Kubernetes Distribution

AKS

Additional Info

No response

[turkenf]

Hi @Crazyigor1987,

Thank you for the issue report, I'm asking to understand fully. Are you saying that the KubernetesCluster resource that you manage in provider-azure-containerservice v1.1.0 does not work after you upgrade the provider version to 1.3.0?

[turkenf]

Are you experiencing this problem only for the KubernetesCluster resource?

[Crazyigor1987]

Hi @turkenf ,

I've updated the providers to v1.4.0. Experience the problem also on this version. These CRDs are not accessable:

diskencryptionsets.compute.azure.upbound.io:
Error from server: conversion webhook for compute.azure.upbound.io/v1beta1, Kind=DiskEncryptionSet failed: Post "https://provider-azure-compute.crossplane.svc:9443/convert?timeout=30s": EOF

dnszones.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=DNSZone failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF

keys.keyvault.azure.upbound.io:
Error from server: conversion webhook for keyvault.azure.upbound.io/v1beta1, Kind=Key failed: Post "https://provider-azure-keyvault.crossplane.svc:9443/convert?timeout=30s": EOF

kubernetesclusternodepools.containerservice.azure.upbound.io:
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesClusterNodePool failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

kubernetesclusters.containerservice.azure.upbound.io:
Error from server: conversion webhook for containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster failed: Post "https://provider-azure-containerservice.crossplane.svc:9443/convert?timeout=30s": EOF

subnets.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=Subnet failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF

vaults.keyvault.azure.upbound.io:
Error from server: conversion webhook for keyvault.azure.upbound.io/v1beta1, Kind=Vault failed: Post "https://provider-azure-keyvault.crossplane.svc:9443/convert?timeout=30s": EOF

virtualnetworks.network.azure.upbound.io:
Error from server: conversion webhook for network.azure.upbound.io/v1beta1, Kind=VirtualNetwork failed: Post "https://provider-azure-network.crossplane.svc:9443/convert?timeout=30s": EOF