Fix issue 276: Double encoded KubernetesCluster connection secrets

[svscheg]

Description of your changes

Fixes first part for issue #276

I have:

• Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

Manually
Uptest: https://github.com/upbound/provider-azure/actions/runs/5268824377

Secret before the changes:
before_fix_changes.txt

Secret after the changes:
after_changed.txt

[ulucinar]

Looks like the kube_admin_config[].client_certificate has already set Sensitive to true but when you remove the configuration override here, a corresponding field gets generated in the status. I was not expecting this, i.e., I was thinking that the configuration override we have here was unnecessary.

Following fields are not marked as "sensitive" in Terraform cli schema output.

We will need to check further why we need this override. The comment here says that these are not marked as sensitive in the JSON schema. But in the native schema, this is already marked as sensitive. So, I believe for some reason, we are losing this important information by not consuming the native schema.

[turkenh]

@svscheg as I stated in the issue, I would suggest using AdditionalConnectionDetailsFn configuration to end up with the exact connection secret we want, e.g. encoded only once.

[turkenh]

See this as an example: https://github.com/upbound/provider-gcp/blob/04f2e8f762a06821509a95fa33c1db4b2f313bde/config/container/config.go#L61

[svscheg]

/test-examples="examples/containerservice/kubernetescluster.yaml"

[turkenh]

@svscheg thanks!

It would be good to check the content of the connection secret after the resource is ready. Could you update the How this code has been tested section with the content of that secret (should be ok to share even sensitive given it is a test cluster to be deleted)?
I am also wondering if you could use that kubeconfig to connect to the cluster with kubectl?

[svscheg]

Hi @turkenh , @turkenf the PR is ready for review, could you please take a look it?

[svscheg]

@svscheg thanks!

It would be good to check the content of the connection secret after the resource is ready. Could you update the How this code has been tested section with the content of that secret (should be ok to share even sensitive given it is a test cluster to be deleted)? I am also wondering if you could use that kubeconfig to connect to the cluster with kubectl?

Hi @turkenh ! I added files with secret before the changes and after the changes.
and also have a question - how I can to check if I can use that kubeconfig to connect to the cluster with kubectl?

[turkenh]

and also have a question - how I can to check if I can use that kubeconfig to connect to the cluster with kubectl?

1. Copy the value of the kubeconfig key in the secret
2. Decode it with:

base64 -D <<< `paste-the-kubeconfig-here` > /tmp/k

3. export KUBECONFIG with

export KUBECONFIG=/tmp/k

4. Try a kubectl command

kubectl get nodes

If the kubectl command works, then we're good.

[svscheg]

and also have a question - how I can to check if I can use that kubeconfig to connect to the cluster with kubectl?

1. Copy the value of the kubeconfig key in the secret
2. Decode it with:

base64 -D <<< `paste-the-kubeconfig-here` > /tmp/k

3. export KUBECONFIG with

export KUBECONFIG=/tmp/k

4. Try a kubectl command

kubectl get nodes

If the kubectl command works, then we're good.

Hi @turkenh !
If I use value from the kubeconfig attribute and repeat all the steps I can see next after command kubectl get nodes:
kubectl get nodes
E0623 19:36:07.366367 8815 memcache.go:238] couldn't get current server API group list: Get "https://exampleaks1-fsl39vxh.hcp.westeurope.azmk8s.io:443/api?timeout=32s": dial tcp 20.103.162.20:443: i/o timeout
E0623 19:36:37.368250 8815 memcache.go:238] couldn't get current server API group list: Get "https://exampleaks1-fsl39vxh.hcp.westeurope.azmk8s.io:443/api?timeout=32s": dial tcp 20.103.162.20:443: i/o timeout
E0623 19:37:07.392539 8815 memcache.go:238] couldn't get current server API group list: Get "https://exampleaks1-fsl39vxh.hcp.westeurope.azmk8s.io:443/api?timeout=32s": dial tcp 20.103.162.20:443: i/o timeout
E0623 19:37:37.395415 8815 memcache.go:238] couldn't get current server API group list: Get "https://exampleaks1-fsl39vxh.hcp.westeurope.azmk8s.io:443/api?timeout=32s": dial tcp 20.103.162.20:443: i/o timeout
E0623 19:38:07.396277 8815 memcache.go:238] couldn't get current server API group list: Get "https://exampleaks1-fsl39vxh.hcp.westeurope.azmk8s.io:443/api?timeout=32s": dial tcp 20.103.162.20:443: i/o timeout
Unable to connect to the server: dial tcp 20.103.162.20:443: i/o timeout

Is it correct or something is not ok?

[svscheg]

@turkenh maybe we have some ways to re-check attributes kubeconfig.clustercacertificate, kubeconfig.clientcertificate, kubeconfig.clientkey?

[turkenh]

Is it correct or something is not ok?

@svscheg, looks like it is a valid kubeconfig but your cluster was not ready yet or there were some connectivity problems. Ideally, I would like to see a successful output with a list of nodes.

@turkenh maybe we have some ways to re-check attributes kubeconfig.clustercacertificate, kubeconfig.clientcertificate, kubeconfig.clientkey?

The above procedure would validate all together without checking them individually.

[turkenh]

This looks good to me.

As I commented above, seeing a successful kubectl get nodes with the generated config would be good before merge. But I am fine with approving as the code itself looks good and previous test @svscheg partially validated that generated kubeconfig is a valid one.

[turkenf]

Tested manually and kubectl get nodes command works properly:

> kubectl get nodes

NAME                              STATUS   ROLES   AGE   VERSION
aks-default-47539582-vmss000000   Ready    agent   19m   v1.25.6
aks-example-21932893-vmss000000   Ready    agent   11m   v1.25.6