-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AKS clusters fail to create - they now need a verified domain of the organization #298
Comments
I too am running in to this issue. It might be important to note that currently this is only associated with using Single Tenant app registration. However, Azure will start restricting Multi-Tenant app registrations Nov. 9th. |
I too a running into this issue. This does seem related to potential changes in Azure's Graph API. We've been running an older version of Crossplane (v.0.14.0). Even though all processes (Crossplane and Azure provider) are still on that version, we receive the same error which halts the creation of the AKSCluster resource. Also confirmed this issue with the latest version of Crossplane and Azure-Provider |
Looks like the identifier URI that we pass here is no longer a valid one due to the recent changes mentioned above. |
According to Azure AD application registration security best practices docs, if we use the However, the However, my observation is that the Azure API behaves differently apart from now doing domain name verification on App ID URIs. While debugging, I have had success in deploying an AKS cluster with that change but when I run the provider, initial attempts of role assignments to the provisioned service principal fail stating that the target service principal does not exist. That principal is provisioned by the provider here. And the If I properly time application/service principal/role assignment/cluster provisioning via debugging, the cluster can successfully be provisioned. I assume this eliminates any issues with the This looks more like a race condition in the provider to me. However, given the high frequency I observed in a small (4-5) number of trials, I do suspect that, apart from the domain validation of identifier URIs, there are some further behavior changes on the Azure side (specifically in the Microsoft graph/authorization services) that made the current implementation susceptible to race conditions. One theory is that we update the application's secret after a successful AKS cluster I have also observed some cases in which the provider is stuck trying to provision the role assignment, which has in fact been already provisioned. In fact, the provider first checks for the role assignment and looks like it does not attempt to assign the role to the service principal if the assignment already exists, so these errors were not expected. Does this mean that |
I suspect, to prevent the seemingly increased-frequency race condition on the application secret, we could store the generated shared secret in a K8s secret and reuse it in subsequent reconciliations. We have done something similar to address a PostgreSQL issue previously. But not sure about the HTTP 409's we get for the role assignments (described above). But maybe that's a temporary one, meaning that eventually the cc. @negz, @sergenyalcin |
@ulucinar could you please share how can we make the change to api instead of https as a temporary workaround? I'm stuck with the creation of new clusters. Thanks! |
@haarchri shared this link in the community meeting that may be useful about the breaking Azure change that is causing this: |
Hi @e9169, url := fmt.Sprintf("api://%s.aks.crossplane.io", name) , and rebuilding the provider. As discussed in the above comment, this addresses the initial problem of the REST API returning an HTTP 400 stating that custom domain verification fails. However, it looks like we have some other issues to be addressed as discussed above. If you get a chance to try it, I'd love to hear about your results. With that change, I was able to provision clusters while debugging the provider but when running it, I hit other issues, which I'd like to investigate. |
Would it be possible to enable custom domains in the call aks.go does? The problem is that the uri being sent is like https://cluster.aks.azure.crossplane.io or something like that and obviously that don't exist. But for those who have a custom domain in place could be as simple as editing a field in the yaml if this was enabled. |
I made the change, built the provider but it is not generating anything inside cluster/local apart from a testing script. The helm chart is not created although the docker images are being created. I pushed the image to minikube and tried to modify the deployment to use my build but it keeps replacing that image with the stable one, I guess it's because of this package:
|
Hi @e9169, |
- Fixes crossplane-contrib#298 Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
- Fixes crossplane-contrib#298 Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
- Fixes crossplane-contrib#298 Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
What happened?
When creating an AKS cluster, the following events are recorded in the
akscluster
description.How can we reproduce it?
The output that follows is from
kubectl get akscluster xyz -o yaml
. It can be used to reproduce the issue.What environment did it happen in?
Azure
kubectl version
)AKS
macOS
uname -a
)The text was updated successfully, but these errors were encountered: