Use Karpenter on CAPZ

[QuantumEnigmaa]

Karpenter can be used on Azure using this upstream repo : https://github.com/Azure/karpenter-provider-azure in self hosted mode.

We should thus adapt the karpenter-bundle app to make it work on CAPZ.

[QuantumEnigmaa]

In our GS environment, karpenter is deployed using the karpenter-bundle app which in turns deploy 3 apps :

• karpenter-app
• capa-karpenter-taint-remover
• karpenter-crossplane-resources

So the first step to have it working on CAPZ is to know which of these apps work out-of-the-box in a CAPZ environment (if there are any) and then adapt those that are not accordingly.

• After deploying the capa-karpenter-taint-remover app on glean, I noticed that it worked perfectly without having to update the app itself, so nothing to do here except renaming the app.
• Issues are arising when it comes to the karpenter-crossplane-resources app. Indeed this app is deploying AWS specific resources such as IAM and SQS related ones. Therefore, this app definitely needs to be updated to support CAPZ deployment;
• Concerning karpenter-app, as the app relies on a <cluster_id>-crossplane-config configmap to be deployed but crossplane is not deployed on glean, there's no such cm in the cluster and thus I couldn't deploy the app. However by removing the extraConfig field related to this one in the app CR it deployed nicely even though the pod was crashlooping (which was expected as it wasn't running in the adequate environment as of the time of testing).

So the remaining things to do are :

• Rename capa-karpenter-taint-remover app to capi-karpenter-taint-remover as well as all other CAPA mentions inside the app's repo.
• Update the karpenter-crossplane-resources with the resources required to have Karpenter running on Azure
  • Find which custom resources are needed for karpenter on Azure.
• Deploy the karpenter-bundle app on glean using the standard procedure (nodepool + provisionner creation) and make it work.