Add Support for serial port as alternative to ssh

[ScheererJ]

What would you like to be added:
gardenctl should provide a convenient way to connect to a virtual machine in the cluster node network.

Why is this needed:
Some cluster owners may configure their networks in a way that ssh is impossible as ingress traffic may be blocked. Most infrastructures support a serial port/console as alternative means to connect to a virtual machine. As the network path goes through the infrastructure it may be able to reach into such highly isolated clusters.
Serial port/console may require a bastion with a provisioned user/password. However, it does not require a virtual machine to be reachable through the internet.

[petersutter]

Some cluster owners may configure their networks in a way that ssh is impossible as ingress traffic may be blocked.

could you give more details? because if ssh is enabled in the shoot spec (Shoot.spec.provider.workerSettings.sshAcces), ingress traffic will be allowed for the bastion. Or is there a case where this is not possible / can be restricted by the user even if ssh is enabled? See also discussion in #325

Most infrastructures support a serial port/console as alternative means to connect to a virtual machine.

E.g. for aws are you referring to ec2 instance connect? https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-methods.html / https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-with-ec2-instance-connect-endpoint.html

I tried it out by first creating an instance-connect-endpoint

$ aws ec2 create-instance-connect-endpoint --subnet-id sg-0xxxxx`

and then

ssh -i $TMPDIR/my.id_rsa gardener@10.123.0.123 -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id i-0xxxxxxxxx --max-tunnel-duration 3600'

but the open-tunnel command fails because the infrastructure credential that I used currently does not have the ec2-instance-connect:OpenTunnel permission (it has the permissions as promoted in the dashboard here https://github.com/gardener/dashboard/blob/9ec152068ed6e3d6973b25ff131906e676107fa6/frontend/src/components/Secrets/GSecretDialogAws.vue#L166-L216). Of course we can add this permission to the list, but shoot owners will have to adapt it accordingly.

I have not tested it out on other infrastructures yet, but certainly any infrastructure resources, like the instance-connect-endpoint should be managed by the corresponding extension, similar to like we do it for the bastion feature so that it is ensured that it is automatically cleaned up - if we want to support this.

[ScheererJ]

Unfortunately, I cannot provide details why my ssh attempts failed. Cluster owners control their network and have plenty options to prevent the connection from working.
In my scenarios, it was likely that internet traffic was routed through peered networks with some firewall dropping packets.

I used https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console, which is fairly easy to utilise. Azure supports something similar (https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-overview), but @kon-angelo mentioned that it is a bit more complex to setup. I have not experimented with serial console support on AWS, but I suppose https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html is what I was looking for. It seems different from what you tried as a serial console explicitly does not need network access. Therefore, specifying a subnet seems strange. Nevertheless, it might require additional permissions.

[petersutter]

Nevertheless, it might require additional permissions

yes, this command requires ec2-instance-connect:SendSerialConsoleSSHPublicKey permission.

aws ec2-instance-connect send-serial-console-ssh-public-key --instance-id i-0xxxxxxxxx --serial-port 0 --ssh-public-key file://$TMPDIR/my.id_rsa

Then the following should work

ssh -i $TMPDIR/my.id_rsa.pub i-0xxxxxxxxx.port0@serial-console.ec2-instance-connect.<region>.aws

[kon-angelo]

In general operational efforts are particularly hard in cases where machines are not able to join the cluster. This is usually caused by networking issues that also prevent SSH one way or the other (network isolation, or that cloud-config-downloader and friends can't setup proper SSH access). The current ssh/bastion command is provider agnostic by virtue of setting it's own jumphost but it doesn't help in these aforementioned cases.

We can leverage some provider-specific capabilities to make our lives easier. Serial console access in particular solves the networking isolation issue so it is good to consider but we probably also need a solution for setting up the access (and send-serial-console-ssh-public-key doesn't have a good counterpart AFAIK)

CC: @MrBatschner