Upgrading k3s version to v1.24.3 fails to perform argocd commands

[decipher27]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Production Environment:
K3s Version:
k3s:v1.21.4-k3s1

Cluster Configuration:
Eks clusters having 4 spot nodes.

We noticed our workflows are failing due to this error - "Failed to wait for service account secret: timed out waiting for the condition". Started happening after upgrading k3s to v1.24.3.
After performing the steps updated here, did solve the issue - #9422 (comment)

To Reproduce
Upgrade k3s to 1.24.3 and then this happens during argocd cluster add or argocd app create step

ArgoCD Version
v2.4.2

Logs
time="2022-08-22T08:39:32Z" level=info msg="ClusterRoleBinding "argocd-manager-role-binding" updated"
time="2022-08-22T08:40:02Z" level=fatal msg="Failed to wait for service account secret: timed out waiting for the condition"
time="2022-08-22T08:40:04Z" level=fatal msg="rpc error: code = InvalidArgument desc = application spec for k3s-upgrade3 is invalid: InvalidSpecError: Unable to get cluster: rpc erro
r: code = NotFound desc = cluster "https://onboarding-tenant.atlan.dev/kubernetes/virtualcluster/loft-cluster-loh2p/vcluster-k3s-upgrade3/k3s-upgrade3\" not found"

Please let me know if any more information is required.

[jannfis]

What is the version of your Argo CD CLI please?

[decipher27]

ArgoCD CLI version is v2.3.4

[jannfis]

Please use a CLI either 2.4.x or >= 2.3.7, as those version will contain the fix #9546 which added support for Kubernetes >=1.24

[james-ransom]

Confirmed working 2.3.7

[decipher27]

This worked. But after upgrading i'm getting this error on my cluster when I do an argo list:

❯ argo list
FATA[2022-08-26T18:36:21.979Z] rpc error: code = NotFound desc = the server could not find the requested resource (get workflows.argoproj.io)

[decipher27]

Current argo CD CLI version: 2.3.7
k3s version: v1.24.3-k3s1
ARGO workflows CLI : v3.1.3

[jannfis]

It seems that the cluster doesn't have the proper CRDs installed.

However, I think you should move that to the Argo workflows issue tracker.

[decipher27]

Found this error on argoCD after upgrading the ArgoCD Cli version, hence resources are not getting created.

the server could not find the requested resource

[gauravkcldcvr]

any update on this? Is there any solutions, we are waiting for response to get this fixed

[decipher27]

jannfis Any updates here?

[jannfis]

I'm pretty sure this is a problem with the cluster, not with Argo CD. "The server could not find the requested resource" usually means that the appropriate CRDs are not installed.

[decipher27]

@jannfis Thank you, Found the issue, since we are using latest version of k3s i.e v1.24.3+k3s1, it requires latest CRD's updated with the latest apiVersion.