Hashicorp Vault Signed SSH credentials not working in 21.0.0

[jbouse]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Summary

Unable to use the Hashicorp Vault Signed SSH credential to sign the SSH key to log into servers using zero-trust method. This appears to be a regression from 20.1.0 where this credential continues to work without problem and only fails when using 21.0.0.

AWX version

21.0.0

Select the relevant components

• UI
• API
• Docs

Installation method

kubernetes

Modifications

no

Ansible version

core 2.12.5.post0

Operating system

MacOS

Web browser

Chrome

Steps to reproduce

1. Configure a Machine credential with the Signed SSH Certificate using a configured Hashicorp Vault SIgned SSH credential.
2. Launch a job template assigned with the Machine credential that is expected to be signed by vault.
3. Watch the job fail to connect to the host.

Expected results

Previously with AWX 20.1.0 using the Hashicorp Vault Signed SSH credentials the Job output would show something like:

Enter passphrase for /runner/artifacts/330395/ssh_key_data:
Identity added: /runner/artifacts/330395/ssh_key_data (jbo...@jbouse-MBP16.lan)
Certificate added: /runner/artifacts/330395/ssh_key_data-cert.pub (vault-approle-7b0ecb9e24638474813385e9f848a40ae9a40f055bde812d8b4530c5c6433cea)

The line stating Certificate added: ... indicating the signed certificate from Vault was being passed along with the Private SSH key that had been unlocked with the passphrase (also retrieved from Vault via Hashicorp Vault Secret Lookup). I would expect to see the same behavior with 21.0.0.

Actual results

In the case of AWX 21.0.0 using the Hashicorp Vault Signed SSH credential I am seeing only the following:

Enter passphrase for /runner/artifacts/195/ssh_key_data:
Identity added: /runner/artifacts/195/ssh_key_data (jbo...@jbouse-MBP16.lan)

I am using the approle auth method for AWX to authenticate to Hashicorp Vault 1.10.3. I've configured the role_id and secret_id accordingly for both the Hashicorp Vault Signed SSH and Hashicorp Vault Secret Lookup credentials. All my Machine credentials have the SSH Private Key using the Hashicorp Vault Secret Lookup to retrieve the key so and non-Signed SSH Machine credentials work fine so I know AWX is properly authenticating with Vault. Only the Signed SSH credentials are failing. Further I have executed the same vault ssh -mode=ca ... CLI call duplicating what AWX should be performing using the same approle credentials and I am able to connect into the servers successfully.

Additional information

AWX is deployed into the EKS cluster using the AWX Operator which was deployed via Helm chart. Both 21.0.0 and 20.1.0 were deployed via AWX Operator but only 21.0.0 appears to be exhibiting this regression.

[tmanninger]

That's a known issue:
#12177 (comment)

[Tioborto]

Hello @jbouse,
This behaviour will be fixed with the next release. I made a test with the devel image (#12122) and all work fine for this auth method.

[jbouse]

@Tioborto thanks... I was looking through the code trying to find the breakage and never found that. Glad it was an easy fix. Any ETA by chance on the release?

[nixocio]

@jbouse https://github.com/ansible/awx/releases/tag/21.1.0
Please, can you inform if the issue was fixed?

[jbouse]

@nixocio let me update my deployment and validate.

[jbouse]

@nixocio Yes, the 21.1.0 image is handling the signed SSH certificate properly now as it did in 20.1.0. I am however seeing an issue now when I launch a job that I'm getting thrown the "Something went wrong..." screen while trying to watch the running job. After the job has been completed, I can open up the output but not while it is running. Unsure if this is related to the new image or not yet.

[jbouse]

@nixocio disregard the error... Turns out my browser must have had a bad cache as restarting the browser and re-logging in everything worked fine.

[nixocio]

@jbouse , thanks for your update. I will close this issue then.