Debugging SSH failures

[gabegorelick]

I'm having trouble connecting to the admin container's SSH server. I can connect to the control container and run enable-admin-container. However, when I attempt to ssh into the admin container, I get Connection refused errors.

Is there a way to check that the admin container successfully started? And is there a way to check SSH logs to confirm this isn't a networking issue? (I've double checked that the ingress rules allow this traffic, but you never know.)

In other words, how do you debug Bottlerocket when you can't connect to the admin container?

[webern]

As far as commands to debug the admin container from the control container, I haven't come up with much.

• From the control container you can check your host container settings with apiclient -u /settings | jq '."host-containers"'
• When you attempt to connect to the admin container you can use ssh -vvv to get more information about the failure.

Some further questions may prove helpful:

• Did you provide your ssh key to the admin container container via host-container user-data?
• Or are you using the EC2 key-pair that you launched the instance with?
• Are you using ec2-user as your login to the admin container?

My connection command typically looks like this (when I launch with an EC2 key-pair):

ssh -i "/somewhere/my-ec2-keypair-private-key.pem" "ec2-user@${INSTANCE_IP}"

[webern]

I'll continue with the troubleshooting efforts below but I do think your feature request is reasonable and something that we will consider. I did a quick search through the issues and I don't see a feature request for host-container status or logs through the Bottlerocket API (though the difficulty of getting from a control container session to an admin container session has come up in a few different guises).

---

We can eliminate two possible causes (key issue, and incorrect api settings) with the following procedure:

###
# From developer system with the SSH public key.

PUBKEY="${HOME}/.ssh/id_rsa.pub"
echo '{"ssh":{"authorized_keys":["'"$(cat ${PUBKEY})"'"]}}' | base64 -w 0

###
# From SSM session in control container.

# Stop the admin container if it's running.
apiclient set host-containers.admin.enabled=false

# Apply the user data, ensure we're running a version of the admin container that supports it, and enable it.
apiclient set \
  host-containers.admin.source="328549459982.dkr.ecr.us-west-2.amazonaws.com/bottlerocket-admin:v0.6.0" \
  host-containers.admin.user-data="<BASE64 DATA>" \
  host-containers.admin.enabled=true

###
# From developer system with the SSH key.
ssh -i "${HOME}/.ssh/id_rsa" "ec2-user@${INSTANCE_IP}"

If that doesn't work, and the the tcp connection is working, then I think we're left with getting the instance console output and looking for system errors. For example (or through the UI):

aws ec2 get-console-output --instance-id "${INSTANCE_ID}"

[gabegorelick]

Stopping and restarting the admin container as specified above seems to have worked, although I don't understand why enable-admin-container doesn't work. I'll try to investigate more and post a bug report if I find anything.

[gabegorelick]

I think I figured it out and it is not exactly a bug. Restarting the admin container wasn't what was fixing the issue. It was downgrading from bottlerocket-admin:v0.7.0 to bottlerocket-admin:v0.6.0 that fixed it.

What changed in v0.7.0? I think the relevant change is bottlerocket-os/bottlerocket-admin-container#20, which ensures that sshd is not started if there are no configured public keys. It turns out I actually didn't have any keys configured (mea culpa). Adding keys fixed it, but it's definitely somewhat surprising that a public key is required for the ssh server to start (although it makes sense in hindsight).

In terms of potential improvements to Bottlerocket, start_admin_sshd.sh dutifully logs errors when it encounters an issue, but there's no great way that I know of to see those when running enable-admin-container from the control container. So again, I think somehow streaming some feedback that the admin container successfully or unsuccessfully started (along with any errors it encountered) back to the control container would really help.

Barring that, documentation for how to capture admin container logs would be useful for installs that already have log shipping set up but are encountering issues with the admin container. In other words: what logs should you be capturing ahead of time so that you can diagnose issues with the admin container should you find yourself locked out?

[webern]

I'm glad you got it working. You are correct that we changed things such that the admin container fails to start sshd in the absence of keys and we agree that this is surprising. We will make a change so that the admin container will start and sshd will run with no keys. That way, at least, you will see "access denied". When you run ssh -vvv you will then see the key negotiation failure and realize what the problem is.

Providing a way to check the health of host containers and get information about why they might have failed seems like like a good idea. Most likely this would be in the form of additional GET routes with the API (either host-container specific, or possibly a call to logdog). Would you like to open an issue for this? I can do so if you'd rather not. Thanks.

[gabegorelick]

@webern Probably best if you open the issues. I'm sure you're much better at capturing all this in a way that's helpful to the project.

what logs should you be capturing ahead of time so that you can diagnose issues with the admin container should you find yourself locked out?

FYI, for this I've started capturing the journal logs from the host-containers@admin.service unit. I think that should be enough to diagnose this type of issue in the future, assuming I don't get it so borked that I can't run the log shipping containers 😄. Might be useful for others as well.

Thanks again for all your help!