feature: support for specifying checks in native nomad services

[shoenig]

In v1.3 Nomad added built-in support for native service discovery. One of the requested features to be built on top of that is support for service checks on services registered with Nomad's service provider.

#12589

Add support for specifying service.checks for Nomad services. The initial implementation will support a subset of the check stanza. We can expand feature support going forward - just getting checks working is a lot of plumbing!

check {
  address_mode = "host|bridge|alloc" 
  name         = "<auto>|custom"
  port         = "<auto>|label"
  type         = "http|tcp"
  path         = "<required for http>"
  protocol     = "http|https"
  timeout      = "<required>"
  interval     = "<required>"
  on_update    = "ignore|require_healthy"
}

Nomad Clients (not Servers yet) will expose an API endpoint for getting the latest check result information related to a specific allocation, e.g.

➜ nomad operator api /v1/allocation/ab43526e-412f-59e7-b6cc-7dfa0d06aedb/checks  | jq .
{
  "d59551afe7cb27bf8a86908909b238aa": {
    “Task”: “task” // task level checks only; absent when unset
    "Check": "one",
    "Group": "fake.fake[0]",
    "ID": "d59551afe7cb27bf8a86908909b238aa",
    "Mode": "healthiness",
    "Output": "nomad: http ok", // truncated on success
    “StatusCode”: 200, // http only; absent when unset
    "Service": "njfakeservice",
    "Status": "success", // or “failing” or “pending”
    "Timestamp": 1657216606
  },
  "945c45e2f572e2cd9f4395e820d2073f": {
    "Check": "two",
    "Group": "fake.fake[0]",
    "ID": "945c45e2f572e2cd9f4395e820d2073f",
    "Mode": "healthiness",
    "Output": "nomad: tcp ok",
    "Service": "njfakeservice",
    "Status": "success",
    "Timestamp": 1657216606
  }
}

The primary use case for check's in Nomad services is for controlling the rollout of deployemnts. Like with Consul services, Nomad will respect on_update to decide whether to treat checks like health checks or readiness checks. If a check is a health check and is failing during a deployment, the deployment does not progress, just as in the case of an analogous Consul check.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.