Blackbox Exporter seems to ignore Host Header for HTTP probe

[stewartn782]

Host operating system: output of uname -a

Running Blackbox Exporter container on ECS

Linux ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal 4.14.209-160.339.amzn2.x86_64 #1 SMP Wed Dec 16 22:44:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.21.1 (branch: master, revision: d26fcc665935e962ba9ce331d3dc82babe0627c4)
  build user:       root@1e0b3966ba2f
  build date:       20220618-12:55:36
  go version:       go1.18.3
  platform:         linux/amd64

What is the blackbox.yml module config.

modules:
  http_2xx:
    prober: http
    timeout: 15s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      headers:
        Host: my.super.host
      follow_redirects: true
      tls_config:
        insecure_skip_verify: true
      preferred_ip_protocol: "ip4" # defaults to "ip6"
      ip_protocol_fallback: false  # no fallback to "ip6"

What is the prometheus.yml scrape config.

  - job_name: 'blackbox-https'
    scrape_timeout: 15s
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
     - targets:
         - https://my.super.host/actuator/health
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: blackbox.ecs.super.host:9115

What logging output did you get from adding &debug=true to the probe URL?

ts=2022-06-22T09:51:51.454000732Z caller=main.go:352 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Beginning probe" probe=http timeout_seconds=15
ts=2022-06-22T09:51:51.454213407Z caller=http.go:336 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Resolving target address" target=my.super.host ip_protocol=ip4
ts=2022-06-22T09:51:51.597195547Z caller=http.go:336 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Resolved target address" target=my.super.host ip=IP_ADDR
ts=2022-06-22T09:51:51.597315207Z caller=client.go:252 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Making HTTP request" url=https://IP_ADDR/actuator/health host=my.super.host
ts=2022-06-22T09:51:51.608998705Z caller=main.go:144 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Received HTTP response" status_code=403
ts=2022-06-22T09:51:51.609091806Z caller=main.go:144 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Response timings for roundtrip" roundtrip=0 start=2022-06-22T09:51:51.59739594Z dnsDone=2022-06-22T09:51:51.59739594Z connectDone=2022-06-22T09:51:51.598781347Z gotConn=2022-06-22T09:51:51.605693494Z responseStart=2022-06-22T09:51:51.608930359Z tlsStart=2022-06-22T09:51:51.598839261Z tlsDone=2022-06-22T09:51:51.605605554Z end=2022-06-22T09:51:51.609079507Z
ts=2022-06-22T09:51:51.609170723Z caller=main.go:352 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Probe succeeded" duration_seconds=0.155050196

What did you do that produced an error?

I have added a new scrape target to the configuration which uses the http_2xx module to do http checks.

What did you expect to see?

I expected the hostname rather than the IP to be used in the url:

ts=2022-06-22T09:51:51.597315207Z caller=client.go:252 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Making HTTP request" url=https://my.super.host/actuator/health host=my.super.host

What did you see instead?

The IP is used in the url rather than the hostname even though it has been specified in the "headers" section of the Blackbox Exporter config. The request seems to be failing with a 403 because the host header isn't being passed through to the service when it makes the call using the IP:

ts=2022-06-22T09:51:51.597315207Z caller=client.go:252 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Making HTTP request" url=https://IP_ADDR/actuator/health host=my.super.host
ts=2022-06-22T09:51:51.608998705Z caller=main.go:144 module=http_2xx target=https://my.super.host/actuator/health level=info msg="Received HTTP response" status_code=403

Using curl works as expected:

[root@hostname ec2-user]# curl -X GET -k -H "Host: my.super.host" https://IP_ADDR/actuator/health
{"status":"UP"}

Removing the host header replicates the issue which Blackbox Exporter is encountering:

[root@hostname ec2-user]# curl -X GET -k https://IP_ADDR/actuator/health
{"message":"Forbidden"}

[badhop]

Have the same problem. Tried versions 0.17.0 and 0.21.1 with the same results.
Try to execute http health check trought different load balancers. Successfully execute checks by setting header Host using curl on the same host, postman, etc. Receive 404 when using blackbox. If remove Host from headers receive 200. balancer1.fqdn could be resolved from host.

  http_2xx_balancer1:
    http:
      headers:
        Accept: '*/*'
        Host: balancer1.fqdn
      method: GET
      preferred_ip_protocol: ip4
      valid_http_versions:
      - HTTP/1.1
      - HTTP/2.0
      valid_status_codes: []
    prober: http
    timeout: 10s

[mmpetarpeshev]

I have the similar issue, the blackbox return 404 for working endpoints , it fails with following log :

`Logs for the probe:
ts=2022-09-07T13:05:32.068871581Z caller=main.go:320 module=http_2xx target=https://my.url.com level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2022-09-07T13:05:32.069504382Z caller=http.go:335 module=http_2xx target=https://my.url.com level=info msg="Resolving target address" ip_protocol=ip4
ts=2022-09-07T13:05:32.079432156Z caller=http.go:335 module=http_2xx target=https://my.url.com level=info msg="Resolved target address" ip=my-lb-ip
ts=2022-09-07T13:05:32.079534865Z caller=client.go:251 module=http_2xx target=https://my.url.com level=info msg="Making HTTP request" url=https://my-lb-ip host=my.url.com
ts=2022-09-07T13:05:32.087609003Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Received HTTP response" status_code=404
ts=2022-09-07T13:05:32.087658114Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=404
ts=2022-09-07T13:05:32.087953971Z caller=main.go:130 module=http_2xx target=https://my.url.com level=error msg="Invalid HTTP version number" version=HTTP/2.0
ts=2022-09-07T13:05:32.088058603Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Response timings for roundtrip" roundtrip=0 start=2022-09-07T13:05:32.079624092Z dnsDone=2022-09-07T13:05:32.079624092Z connectDone=2022-09-07T13:05:32.08060517Z gotConn=2022-09-07T13:05:32.085857596Z responseStart=2022-09-07T13:05:32.08755494Z tlsStart=2022-09-07T13:05:32.080633552Z tlsDone=2022-09-07T13:05:32.085654131Z end=2022-09-07T13:05:32.087935858Z
ts=2022-09-07T13:05:32.088584557Z caller=main.go:320 module=http_2xx target=https://my.url.com level=error msg="Probe failed" duration_seconds=0.019597321

Metrics that would have been returned:
probe_dns_lookup_time_seconds 0.009964898
probe_duration_seconds 0.019597321
probe_failed_due_to_regex 0
probe_http_content_length 19
probe_http_duration_seconds{phase="connect"} 0.000981106
probe_http_duration_seconds{phase="processing"} 0.001697285
probe_http_duration_seconds{phase="resolve"} 0.009964898
probe_http_duration_seconds{phase="tls"} 0.005020672
probe_http_duration_seconds{phase="transfer"} 0.000380997
probe_http_redirects 0
probe_http_ssl 1
probe_http_status_code 404
probe_http_uncompressed_body_length 19
probe_http_version 2
probe_ip_addr_hash 7.88769462e+08
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1.694091712e+09
probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
probe_ssl_last_chain_info{fingerprint_sha256="ae66cb45d40024051b6d42a3f46ea6ab67148ce37115bb223aba219f2e1da873"} 1
probe_success 0
probe_tls_version_info{version="TLS 1.3"} 1

Module configuration:
prober: http
timeout: 5s
http:
valid_http_versions:
- HTTP/1.1
- HTTP/2
preferred_ip_protocol: ip4
ip_protocol_fallback: true
tls_config:
insecure_skip_verify: true
follow_redirects: true
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
dns:
ip_protocol_fallback: true`

As curl and browsers are working without any issues.
We are using helm charts and I tried all possible configs for blackbox and prometheus scrape.