"nomad logs" with TLS broken

[andreynpetrov]

Nomad version

v0.5.2

Operating system and Environment details

Ubuntu 16.04 x86_64

root@node-2:~# env | grep NOMAD
NOMAD_CACERT=/etc/pki/tls/certs/ca.pem
NOMAD_CLIENT_CERT=/etc/pki/tls/certs/nomad.pem
NOMAD_CLIENT_KEY=/etc/pki/tls/private/nomad-key.pem
NOMAD_ADDR=https://localhost:4646

Issue

Get error on "nomad logs":
root@node-2:~# nomad logs 8095d822
Error reading file: Get https://172.28.128.8:4646/v1/client/fs/logs/8095d822-b270-bcdc-d7fc-2db37f69d109?follow=false&offset=0&origin=start&task=nginx&type=stdout: x509: cannot validate certificate for 172.28.128.8 because it doesn't contain any IP SANs

But "nomad status" works:
root@node-2:~# nomad status
ID Type Priority Status
test service 50 running

Reproduction steps

Enable TLS and setup env:
export NOMAD_ADDR=https://localhost:4646
export NOMAD_CACERT=/etc/pki/tls/certs/ca.pem
export NOMAD_CLIENT_CERT=/etc/pki/tls/certs/nomad.pem
export NOMAD_CLIENT_KEY=/etc/pki/tls/private/nomad-key.pem

[dbresson]

I believe the problem is the advertised http address on the server running the allocation. You're advertising 172.28.128.8 instead of localhost. This should be fixed by adding the advertise directive to your agent config.

[schmichael]

@dbresson Advertising localhost is almost never correct (unless you're just testing a "cluster" locally).

I think there's a bug in our TLS configuration when the client tries to connect to the node running the allocation.

@andreynpetrov Is the certificate on 172.28.128.8 (should be a client node) valid for the name nomad.${REGION}.client? If so you've got things configured correctly and our client is doing the certificate validation wrong.

Adding -tls-skip-verify should workaround this bug until I get it fixed.

[schmichael]

You can check the names a certificate is valid for with:

openssl x509 -in $NOMAD_CLIENT_CERT -noout -text

And look for the Subject and Subject Alternative Name

Alternatively with cfssl and jq:

cfssl-certinfo -cert $NOMAD_CLIENT_CERT | jq -r '.subject.names+.sans|join("\n")'

[andreynpetrov]

openssl output:
X509v3 Subject Alternative Name:
DNS:server.global.nomad, DNS:client.global.nomad, DNS:localhost

In client config:
advertise {
http = "172.28.128.8"
rpc = "172.28.128.8"
serf = "172.28.128.8"
}

Maybe I need to advertise hostname and add it to cert?

[schmichael]

@andreynpetrov That would fix it, but that's not how we intend mutual TLS auth in Nomad to work.

The intention is that your nodes are more or less ephemeral and so generating new certificates with the proper hostname for every new node is untenable.

Instead we intend people to do exactly what you did: use {role}.{region}.nomad as the name (where role is client or server). That way you can use Packer or Terraform or something to just have the client certificate available on any new nodes you spin up.

So this is a bug, but you can work around it with -tls-skip-verify or the solution you mention.

[andreynpetrov]

Good to know. Thank you for clarification.

[dbresson]

Sorry for the misinformation, I was unaware of the oddball way nomad does tls cert verification.

[schmichael]

@dbresson It's something we may be changing in the future to be a bit less oddball while still not requiring unique certificates per-host.

[ashald]

Same issue appears with fs command:

$ nomad fs 190ff0a5-c8f6-ba79-dfa5-b6b668368b94
Get https://10.10.1.2:4646/v1/client/fs/stat/190ff0a5-c8f6-ba79-dfa5-b6b668368b94?path=%2F: x509: cannot validate certificate for 10.10.1.2 because it doesn't contain any IP SANs

@schmichael asked me to mention it here

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.