-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create ArgoCD cluster with declarative setup using existing secret containing kubeconfig #4651
Comments
It seems strange to reference secret from another secret. Can you please clarify why don't create argocd compatible secret? Do you need to share one secret with Kubeconfig between Argo CD and some other tool? |
Yes, that kubeconfig secret is created/managed by another tool: The new cluster is created dynamically from custom resources. After its creation, the "Cluster API" controller (which created the cluster) creates a new secret which contains just the kubeconfig. |
Thank you for the information @MPV ! So config is created by Cluster API. In the past, we've made changes to get smooth integration with other opensource project. E.g. changed secret key to How do you think, is it possible to do something similar in this case? E.g. Argo CD might support a new secret like |
I'm looking for something like what is shown in this video (which I already have working), but with the addition of automatically deploying apps into the newly created cluster, just after its creation: ...and without needing to push more than one git commit, containing all of:
|
@alexmt Yeah, let's investigate. I opened up a mirror issue over at kubernetes-sigs/cluster-api#3866 |
I'd love to see this too. Currently you have to jump through hoops to make this work, e.g. https://github.com/janwillies/crossargo-sync, https://github.com/infracloudio/crossplane-secrets-controller, ... It would be very convenient if argocd would take a regular kubeconfig |
For reference, this is how such a secret looks like (in the Cluster-API example): $ kubectl -n mycluster get secret mycluster-1-kubeconfig -o yaml
apiVersion: v1
data:
value: obfuscated-base-64-encoded
kind: Secret
metadata:
creationTimestamp: "2020-10-23T14:52:51Z"
labels:
cluster.x-k8s.io/cluster-name: mycluster
name: mycluster-1-kubeconfig
namespace: mycluster
ownerReferences:
- apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
blockOwnerDeletion: true
controller: true
kind: KubeadmControlPlane
name: mycluster-kubeadm-control-plane-1
uid: aa9353fd-25b6-49b4-afa7-c7924191baa1
resourceVersion: "779511"
selfLink: /api/v1/namespaces/mycluster/secrets/mycluster-1-kubeconfig
uid: <obfuscated>
type: Opaque |
#4600 talks about making ExecProvider available in the Cluster config. Maybe it could become a way/workaround to solve this (by running something that gets the credentials from the existing secret and outputs it to stdout, as per default |
Hi @alexmt! I'm one of the Cluster API maintainers. I'm curious about your statement:
Cluster API is currently creating a secret that contains a kubeconfig. However, this secret is only created after you declaratively request a new cluster (e.g. you create a I think what @MPV is asking for is the ability for Argo CD to recognize, support, and use the Cluster API secret containing the kubeconfig. It sounds like you're perhaps ok with Argo CD supporting a secret containing a kubeconfig, based on what you wrote above? Would you be open to supporting the Cluster API kubeconfig secret? Argo CD would have to accept that the secret will not exist for a period of time, but it will eventually get created as Cluster API reconciles things. |
The task that we would like to automate is this step (https://argoproj.github.io/argo-cd/getting_started/#5-register-a-cluster-to-deploy-apps-to-optional)
ArgoCD shall monitor for secrets that is created by cluster-api. There is a label/annotation that can be used and the secret name ends with I guess it's possible to create a hack with the shell operator that do this, but it would of course be much nicer if ArgoCD had a built in support for this cluster-api kubeconfig secret. |
crossplanes apiVersion: v1
kind: Secret
type: connection.crossplane.io/v1alpha1
metadata:
name: foo
namespace: crossplane-system
ownerReferences:
- apiVersion: eks.aws.crossplane.io/v1beta1
controller: true
kind: Cluster
name: foo
uid: cd04c529-8f91-4a1a-a827-2f6138ec4087
data:
clusterCA: LS0tL...
endpoint: aH...
kubeconfig: YXB... |
guys! i can see only 2 in CRDs list : k get crd | grep arg
applications.argoproj.io 2020-09-24T14:52:28Z
appprojects.argoproj.io 2020-09-24T14:52:29Z there is nothing called |
Hi there, maybe someone cen help me: Is there a way how to
Details: I have the ArgoCD server running and wanna define a In the CLI I could do: I didn't found how to create that I am creating my clusters with Crossplane. Crossplane saves the apiVersion: v1
kind: Secret
metadata:
name: cluster-details-my-cluster
namespace: default
uid: 50c7acab-3214-437c-9527-e66f1d563409
resourceVersion: '12868'
creationTimestamp: '2022-01-06T19:03:09Z'
managedFields:
- manager: crossplane-civo-provider
operation: Update
apiVersion: v1
time: '2022-01-06T19:03:09Z'
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:kubeconfig: {}
f:type: {}
selfLink: /api/v1/namespaces/default/secrets/cluster-details-my-cluster
data:
kubeconfig: >-
YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFJFTkRRVkl5WjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVdwTlUwVjNTSGRaUkZaUlVVUkVRbWh5VFROTmRHTXlWbmtLWkcxV2VVeFhUbWhSUkVVeVRrUkZNRTlVVlROT1ZFbDNTR2hqVGsxcVNYZE5WRUV5VFZScmQwMXFUWGxYYUdOT1RYcEpkMDFVUVRCTlZHdDNUV3BOZVFwWGFrRnFUVk5_SHORTENED
type: Opaque
The apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHd_SHORTENED
server: https://MY.IP.TO.K8S:6443
name: my-cluster
contexts:
- context:
cluster: my-cluster
user: my-cluster
name: my-cluster
current-context: my-cluster
kind: Config
preferences: {}
users:
- name: my-cluster
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJQS9adEZFT1Avcnd3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOalF4TkRrMU56VXlNQjRYRFRJeU1ERXdOakU1TURJek1sb1hEVEl6TURFdwpOakU1TURJek1sb3dNREVYT_SHORTENED
client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpJNlVhTDlLem9yL1VpdzlXK1NNUTAxV1BES2ZIK_SHORTENED Thank you very much in advance |
@exocode I don't have a GitOps ready declarative solution for this - currently using Terraform + k8s provider talking to both clusters to do it - but I can point you in the right direction, maybe you'll find a more elegant way.
You can read a bit more about how to add a new cluster to ArgoCD without If you do solve it, please, let us know how. |
I have implemented an experimental operator to synchronize generated secrets to Argo CD cluster secret. It has some way to go and I am not sure if this is the best way to do this. But you can give it a try if it helps. https://github.com/a1tan/argocdsecretsynchronizer |
From what I see the easiest thing would be if the argocd-controller would watch the cluster-api's The other option would be to extend cluster-api to write secrets in the form Argo CD accepts. Would that be an enhancement that the Argo CD codebase accepts? Because it would mean a 'hard' integration into the way cluster-api behaves. Or delegate it to the Argo CD Operator to generate the Argo CD cluster secrets based on the cluster-api secrets. Other than that I think to add the functionality it wouldn't require a lot of effort, so I'd be happy to try myself contributing this, when there is a need for it. |
@Moglum thanks for linking to the inlets blog post Learn how to manage apps across multiple Kubernetes clusters where we gave the example on adding a remote cluster delcaratively. This doesn't seem to work with the newest change in K8s 1.24 where service accounts and tokens are not created in the same way as before.
I'm not sure what the workaround is, but don't know if this PR is related? #9546 |
Edit: Looks like we may have an alternative now for K8s 1.25:
|
Hi together, I also have the case that I like to "automatically" add clusters to ArgoCD that I provisioned with cluster-api using the execProviderConfig from #4600. I managed to get the token from the execProviderConfig (as in the GCP example). The problem I have now is that I need to set server and certificate-authority-data, otherwise ArgoCD will not recognize the cluster. For further testing, I built a local kubeconfig and tried to test that behavior, with the following KubeConfig. For testing I just use cat on a static credentials file:
My credentials file looks like this from the k8s-documentation:
When I run this, I get the following message:
It seems that you can't let the Kubernetes client get hostname and certificate of the api-server from the exec plugin. -> Based on this, I would say that it is not possible to create a generic secret using only the providerExecConfig, which would allow argocd to automatically use a cluster. Please let me know if you have other ideas. |
You might consider using a Kyverno Generate policy to create the ArgoCD cluster secrets as is done here specifically for Rancher CAPI. |
Tangential, but in trying to mimic this behavior using Terraform, I noticed by default the secret name seems to be
|
We managed to achieve automated vcluster registration with a kyverno policy:
Thanks @ron1 for the rancher example |
Hi everyone, I think this problem can be solved using helm lookup function as follows:
But I haven't been able to get it work yet.
Did anyone use the |
Is there a way to give kubeconfig instead of bearer token either secret format or base64 format to register cluster? |
I'm currently trying the same:
The usecase here is to have as few as possible manual steps when bootstrapping a new app-of-apps cluster. |
Close this issue in favor of newer more targeted issues? Since this issue was created, several solutions have arisen:
The following issues are specifically about registration of CAPI clusters: |
Closing in favor of newer more targeted issues. |
Summary
Does declarative setup for ArgoCD already support creating new clusters using an existing secret containing a
kubeconfig
file?If this isn't already supported: this is what I'm after.
Motivation
I'd like to use ArgoCD + Cluster-API for:
Proposal
Allowing
argocd.argoproj.io/secret-type: cluster
secrets to refer to another secret containing akubeconfig
(as part of/instead of it'sconfig
section) and use it for reaching that target cluster.I'm referring to this functionality:
https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#clusters
So instead of:
You could do this:
Or:
The text was updated successfully, but these errors were encountered: