Can't run script health-checks for Consul-connect enabled Nomad jobs.

[zhelezovartem]

Nomad version

Nomad v0.12.5 (514b0d6)

Operating system and Environment details

Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Also tested on Amazon Linux 2 - the same behavior

Issue

I wanted to setup script health-checks for Consul-connect enabled jobs.
I started with simple job with script healthcheck and without consul-connect feature. Check passed. Then I enabled Consul-connect feature for this job and after that health-check become fail.

Reproduction steps

1. Run job without consul-connect - check passed
2. Run job with Consul-connect - check failed

Job file

job file, Consul-connect disabled:

job "example" {
  datacenters = ["eu-central-1"]
  group "cache" {
    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
        port_map {
          db = 6379
        }
      }
      service {
        name = example
        port = "db"
        check {
          type     = "script"
          name     = "test"
          command  = "true"
          interval = "60s"
          timeout  = "5s"
        }
      }
      resources {
        cpu    = 500
        memory = 256
        network {
          mbits = 10
          port  "db"  {}
        }
      }
    }
  }
}

job file, Consul-connect enabled:

job "example" {
  datacenters = ["eu-central-1"]
  group "cache" {
    network {
      mode = "bridge"
    }
    service {
      name = "example"
      port = "6379"
      connect {
        sidecar_service {}
      }
      check {
        type     = "script"
        name     = "test"
        command  = "true"
        interval = "60s"
        timeout  = "5s"
      }
    }
    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
      }
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Nomad Client logs

There is nothing about this health-check in Nomad client logs, but i can see warning in Consul client logs:

[WARN]  agent: Check missed TTL, is now critical: check=_nomad-check-c39120b3a292cd4511306eac4c8e425dfe70872a

Nomad Server logs

There is nothing about this health-check in Nomad server logs

[shoenig]

Hi @zhelezovartem thanks for reporting. Can you try setting the task parameter for the check in the Connect case?

I think when the service is defined at the group level, setting the task is required for script checks - otherwise Nomad does not know which task driver to use to execute the check. Since the check never executes, the result never gets reported to Consul, and Consul complains about missing the TTL.

Nomad should both document and provide validation on job submission to check for this case.

[zhelezovartem]

Hi @shoenig, thanks a lot for great explanation. I followed your advice and now it works. Working job config looks like:

job "example" {
  datacenters = ["eu-central-1"]
  group "cache" {
    network {
      mode = "bridge"
    }
    service {
      name = "example"
      port = "6379"
      connect {
        sidecar_service {}
      }
      check {
        type     = "script"
        task     = "redis"
        name     = "test"
        command  = "true"
        interval = "60s"
        timeout  = "5s"
      }
    }
    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
      }
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

It was a bit unclear for me that check was not executed when inspecting Nomad logs. Maybe will be good to add some sort of warning to Nomad logs for such cases? Something like healthcheck defined but will never be executed, what do you think?

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.