Retrieving stats from CLI doesn't properly support TLS

[schmichael]

I think this is a similar bug as #2490.

Original report by @pznamensky:

I've found that at least nomad alloc-status $alloc-id doesn't work as expected (for me).

~ $ nomad alloc-status e90647d0
ID = e90647d0
Eval ID = f693f405
....

Couldn't retrieve stats (HINT: ensure Client.Advertise.HTTP is set): Get https://$alloc-ip:4646/v1/client/allocation/e90647d0-6dda-545b-4c48-69094ec65b06/stats: x509: certificate is valid for 127.0.0.1, not $alloc-ip

Task "redis" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
500 MHz 256 MiB 300 MiB 0 db: $alloc-ip:27590

Task Events:
...

This doesn't work in case of using cli or client certs.
Is this expected behavior?

[dsolsona]

I'm having the same issue.

Also node-status seems to be affected as well

nomad node-status e7682fec

error fetching node stats (HINT: ensure Client.Advertise.HTTP is set): Get https://172.28.1.138:4646/v1/client/stats: x509: certificate is valid for 127.0.0.1, not 172.28.1.138

[schmichael]

@dsolsona Thanks for the report! The good news its it's the same underlying call in the api package not correctly handling TLS in both cases. Still hoping to fix by 0.6.1.

[pznamensky]

Thanks!

[dsolsona]

Hi @schmichael

I've tested Nomad 0.6.2 and now I'm getting a completely different error

Couldn't retrieve stats (HINT: ensure Client.Advertise.HTTP is set): Get https://10.91.89.3:4646/v1/client/allocation/a02a19be-6824-851c-db49-6297f3138045/stats: x509: certificate is valid for client.aws.nomad, localhost, not client..nomad

It seems that somehow the region is getting lost based on the log output.

On the Nomad client I'm seeing

http: TLS handshake error from 10.91.70.24:48172: remote error: tls: bad certificate

Both servers and clients have the right region in them and the certificates were created with the right region too.

If you think this issue is not the best place to discuss this I can create a new one or move this to the mailing list, but it seems kind of related to the latest Nomad release (0.6.2)

[dadgar]

@dsolsona Can you test this binary:
nomad.zip

You would only need to replace it for the CLI. No need to change server/client binaries.

[dsolsona]

Hi @dadgar

I've tested the binary and now I get

Couldn't retrieve stats (HINT: ensure Client.Advertise.HTTP is set): Get https://10.91.89.3:4646/v1/client/allocation/119c1280-4720-a27e-aac8-18a83faa4385/stats?region=global: x509: certificate is valid for client.aws.nomad, localhost, not client.global.nomad

Still using the wrong region, but at least now it uses a region ;)

[dadgar]

@dsolsona Did you set the region to aws? You have two ways

1. export NOMAD_REGION=aws
2. nomad node-status -region=aws <id>

[dsolsona]

Oh right, exporting NOMAD_REGION does work flawlessly

I guess I was expecting it would read the region from the nomad configuration file.

Thanks for fixing this so quickly!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.