HTTP Healthcheck not sending Host header

[SamMousa]

Nomad version

v1.4.2

Operating system and Environment details

Ubuntu 22.0.4.1

Issue

In a multi-host web app I noticed the status check never turned to healthy. Upon debugging I found that the status endpoint is returned a 404 because there was no Host header.

I then proceeded to configure custom headers like this:

check {
          name = "status endpoint"
          type     = "http"
          path     = "/site/status"
          interval = "5s"
          timeout  = "1s"
          method   = "GET"
          header {
            Host = ["my.host"]
            User-Agent = ["Nomad"]
          }
        }

Note the modified user agent; I added this to confirm that headers are being added. The received request however looks like this:

{
  "timestamp": "2022-11-18T09:18:36+00:00",
  "body_bytes_sent": 2572,
  "request_time": 0.048,
  "response_status": 404,
  "request": "GET /site/status HTTP/1.1",
  "request_method": "GET",
  "host": "192.168.40.3",
  "upstream_cache_status": "",
  "http_user_agent": "Nomad",
  "nginx_access": true
}

Reproduction steps

Configure a health check with custom host header

Expected Result

I expect all the configured headers to be sent

Actual Result

Host header is not sent

[tgross]

Hi @SamMousa! I gave this a try myself and I wasn't able to confirm what you're seeing. Here's the jobspec I used:

jobspec

job "example" {
  datacenters = ["dc1"]

  group "web" {

    network {
      mode = "bridge"
      port "www" {
        to = 8001
      }
    }

    service {
      port = "www"

      check {
        name     = "status endpoint"
        type     = "http"
        path     = "/index.html"
        interval = "5s"
        timeout  = "1s"
        method   = "GET"
        header {
          Host       = ["my.host"]
          User-Agent = ["Nomad"]
        }
      }

    }

    task "http" {

      driver = "docker"

      config {
        image   = "busybox:1"
        command = "nc"
        args    = ["-lvp", "8001"]
        ports   = ["www"]
      }

    }
  }
}

And here's the logs from that allocation:

$ nomad alloc logs 12ad
GET /index.html HTTP/1.1
Host: my.host
User-Agent: Nomad
Accept: text/plain, text/*, */*
Accept-Encoding: gzip
Connection: close

Can you share more of your jobspec, and which version of Consul you're using? Maybe there's a regression between versions here, or I'm missing some other piece of the puzzle.

[SamMousa]

Sorry for not being more explicit. I'm using a nomad native service; no consul

[shoenig]

Seems with the Go net/http client, we need to set the request.Host field for the Host header to actually be set golang/go#29865

[SamMousa]

Fixed in my first ever PR to Nomad and in the Go language.

[tgross]

Fixed by #15337, which will ship in the next regular version of Nomad (likely 1.4.4)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.