service disco: support for tls_skip_verify on nomad https checks

[shoenig]

https://developer.hashicorp.com/nomad/docs/job-specification/check#tls_skip_verify

We should implement this flag for Nomad's native service discovery. Currently running into this myself while trying to use a local Caddy as a mock ACME server. The cert is signed only for localhost but Nomad only knows about the IP address (127.0.0.1) so I can't ever have a passing check due to the tls error.

caddy nomad job

job "caddy" {
  type = "service"

  group "group" {
    network {
      mode = "bridge"
      port "acme" {
        static = 6666
      }
    }

    service {
      name     = "acme"
      port     = "acme"
      provider = "nomad"
      check {
        name     = "alive"
        type     = "http"
        protocol = "https"
        path     = "/"
        interval = "10s"
        timeout  = "1s"
        tls_skip_verify = true
      }
    }

    task "caddy" {
      driver = "raw_exec"

      artifact {
        source      = "https://github.com/caddyserver/caddy/releases/download/v2.6.4/caddy_2.6.4_linux_amd64.tar.gz"
        destination = "local/"
      }

      template {
        data = <<EOH
          localhost:6666 {
            tls internal
            acme_server * {
              ca "local"
	            lifetime "1h"
            }
          }
          # make requests to
          # https://localhost:6666/acme/local/directory        
        EOH

        destination = "local/Caddyfile"
      }

      config {
        command = "local/caddy"
        args    = ["run", "--config", "local/Caddyfile"]
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}

nomad alloc checks b0
Status of 1 Nomad Service Checks

ID         =  9204a81827eeb24e4351132407199f2f
Name       =  alive
Group      =  caddy.group[0]
Task       =  (group)
Service    =  acme
Status     =  failure
Mode       =  healthiness
Timestamp  =  2023-02-17T11:51:03-06:00
Output     =  nomad: Get "https://127.0.0.1:6666/": remote error: tls: internal error

curl equivalent of the Nomad http check

/usr/bin/curl https://127.0.0.1:6666
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

curl https://localhost:6666 works of course

/usr/bin/curl -w '%{response_code}' "https://localhost:6666"
200

[Antse]

hello @shoenig 👋 , any update on that topic ?

[shoenig]

Hi @Antse! We haven't done anything with this issue yet; it should be pretty easy to knock out though, if you need it 🙂

[Antse]

Hi @Antse! We haven't done anything with this issue yet; it should be pretty easy to knock out though, if you need it 🙂

I mitigate this by using tcp check but this is very dirty :(