Health check routine leaks using new nomad provider

[thetooth]

Nomad version

Nomad v1.4.3 (f464aca)

Issue

I have a service that logs HTTP requests and noticed that the endpoint given for health checking is being executed a few hundred times per second. There is a pretty aggressive restart policy on this job and we had a netsplit issue last night which lead to the service restarting around 600 times, so the logs are quite busy to say the least.

Reproduction steps

Run the job below and either stop the job and resubmit or have the process crash. The number of requests hitting the service increase until nomad is restarted.

Job file (if appropriate)

job "signaling" {
  datacenters = ["cloud"]
  type        = "service"

  reschedule {
    unlimited      = true
    delay          = "15s"
    delay_function = "constant"
    attempts       = 0
  }

  group "signaling" {
    restart {
      attempts = 2
      delay    = "1s"
      interval = "15s"
      mode     = "fail"
    }

    volume "opt" {
      type      = "host"
      source    = "opt"
      read_only = true
    }

    network {
      mode = "host"
      port "api" {
        static = 8000
      }
    }

    task "signaling" {
      driver = "exec"

      config {
        command = "entry.bash"
      }

      template {
        data = <<EOH
#!/bin/bash

/local/opt/signaling -bind=:8000
EOH

        destination = "local/entry.bash"
        perms       = "755"
      }

      template {
        data        = <<EOH
{{ with nomadVar "proapps" -}}
PROAPPS_ID={{ .id }}
PROAPPS_SECRET={{ .secret }}
{{- end }}
EOH
        destination = "local/file.env"
        env         = true
      }

      volume_mount {
        volume      = "opt"
        destination = "/local/opt"
      }

      resources {
        cpu    = 100 # Mhz
        memory = 512 # Mb
      }

      service {
        name     = "api"
        port     = "api"
        provider = "nomad"
        check {
          type     = "http"
          path     = "/debug/pprof/"
          interval = "5s"
          timeout  = "1s"
          method   = "GET"
        }
      }
    }
  }
}

[shoenig]

Thanks for the report @thetooth, and apologies for the slow response. I was able to reproduce this with this simpler job and bash script. AFAICT the duplication in requests to the healthcheck happens on reschedule, which TBH is surprising, but at least now I know where to look.

job "demo" {
  datacenters = ["dc1"]

  group "group1" {
    network {
      mode = "host"
      port "http" {
        static = 8888
      }
    }
    
    reschedule {
      unlimited = true
      delay = "15s"
      delay_function = "constant"
      attempts = 0
    }
    
    restart {
      attempts = 2
      delay = "1s"
      interval = "15s"
      mode = "fail"
    }

    task "task1" {
      driver = "raw_exec"
      user = "shoenig"

      config {
        command = "python3"
        args = ["-m", "http.server", "8888", "--directory", "/tmp"]
      }

      service {
        provider = "nomad"
        port = "http"
        check{
          path = "/"
          type = "http"
          interval = "3s"
          timeout = "1s"         
        }
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

#!/usr/bin/env bash

while true
do
  sleep 5
  pid=$(ps -ef | grep http.server | head -n1 | fields 2)
  echo "kill pid $pid"
  kill -9 $pid
done