connect: enable configuring proxy.expose.checks easy button #7396

shoenig · 2020-03-20T15:15:28Z

Part of #6120

Building upon the support for enabling connect proxy paths in #7323, this change adds the ability to set the proxy.expose.checks flag, which will cause Nomad to automatically create expose path configurations necessary for compatible task-group service check definitions. This feature is analogous to what is available in Consul, but the implementation does not make use of Consul's checks flag, due to the necessary plumbing around port mapping in the Nomad Connect integration.

Given this example,

network {
  mode = "bridge"

  port "forchecks" {
    to = -1
  }
}

service {
  name = "myserver"
  port = 2000

  connect {
    sidecar_service {
      proxy {
        expose {
          checks = true
        }
      }
    }
  }

  check {
    name     = "mycheck-myserver"
    type     = "http"
    protocol = "http"
    port     = "forchecks"
    interval = "3s"
    timeout  = "2s"
    method   = "GET"
    path     = "/classic/responder/health"
  }
}

Nomad will automatically inject (via job endpoint mutator) the appropriate expose path configuration, i.e.

expose {
  path {
    path            = "/classic/responder/health"
    protocol        = "http"
    local_path_port = 2000
    listener_port   = "forchecks"
  }
}

Documentation & e2e tests covering this + manual expose paths coming in another PR.

This helps reduce the number of squiggly lines in Goland.

Enable configuration of HTTP and gRPC endpoints which should be exposed by the Connect sidecar proxy. This changeset is the first "non-magical" pass that lays the groundwork for enabling Consul service checks for tasks running in a network namespace because they are Connect-enabled. The changes here provide for full configuration of the connect { sidecar_service { proxy { expose { paths = [{ path = <exposed endpoint> protocol = <http or grpc> local_path_port = <local endpoint port> listener_port = <inbound mesh port> }, ... ] } } } stanza. Everything from `expose` and below is new, and partially implements the precedent set by Consul: https://www.consul.io/docs/connect/registration/service-registration.html#expose-paths-configuration-reference Combined with a task-group level network port-mapping in the form: port "exposeExample" { to = -1 } it is now possible to "punch a hole" through the network namespace to a specific HTTP or gRPC path, with the anticipated use case of creating Consul checks on Connect enabled services. A future PR may introduce more automagic behavior, where we can do things like 1) auto-fill the 'expose.path.local_path_port' with the default value of the 'service.port' value for task-group level connect-enabled services. 2) automatically generate a port-mapping 3) enable an 'expose.checks' flag which automatically creates exposed endpoints for every compatible consul service check (http/grpc checks on connect enabled services).

Similar to a Consul Service Definition, enable users to configure a single boolean which extrapolates proxy expose rules from existing service checks. By setting 'expose.checks', Nomad will automatically generate minimal 'proxy.expose.path' values for compatible service checks for connect-enabled services. In doing so, Consul checks should then Just Work if they are of type 'http' or 'grpc'.

nickethier

This looks great Seth, I just have some questions around the UX and wanted to here your thoughts. I added a few comments inline about it. Also why is the address_mode set to driver in the example?

nickethier · 2020-03-24T14:01:14Z

nomad/job_endpoint_hook_expose.go

+
+	// The Path, Protocol, and PortLabel are just copied over from the service
+	// check definition. It is required that the user configure their own port
+	// mapping for each check, including setting the 'to = -1' sentinel value


Since I understand how the sausage is made I know why the user must do this, but I think from a user's perspective this could be a bit confusing. Could we not just also generate and inject the appropriate port block to make this work for the user?

I chatted with @schmichael about this; the quirky part is we'd need to generate a name for the port based on the content of the generated expose.path fields. While technically possible it opens the door for some worm filled cans that would need to be addressed (e.g. managing generated ports during an in-place update of a check definition). We settled on just requiring a manually configured port map, at least for now. (If there's demand, I'm certainly on board with making the magic work)

nickethier · 2020-03-24T14:07:01Z

jobspec/parse_service.go

-		"path", // an array of path blocks
-		// todo(shoenig) checks boolean
+		"path",   // an array of path blocks
+		"checks", // single boolean


Is it possible that a user would have a service with both checks exposed through the new expose feature and checks that perhaps just bind to a different port and the user just creates a port mapping? Maybe not but I don't want to back ourselves into a corner where this is an all or nothing configuration.

Opinions on instead annotating each check with a field expose = true?

Opinions on instead annotating each check with a field expose = true?

That's not a bad idea IMHO. The reason it's done the way it is was to follow Consul's precedent - which as I understand is more important to stay consistent.

shoenig · 2020-03-24T18:26:09Z

Also why is the address_mode set to driver in the example?

Whoops, that was vestigial from me trying to debug things. Setting driver in the check is not necessary.

…hecks Part of #6120 Building on the support for enabling connect proxy paths in #7323, this change adds the ability to configure the 'service.check.expose' flag on group-level service check definitions for services that are connect-enabled. This is a slight deviation from the "magic" that Consul provides. With Consul, the 'expose' flag exists on the connect.proxy stanza, which will then auto-generate expose paths for every HTTP and gRPC service check associated with that connect-enabled service. A first attempt at providing similar magic for Nomad's Consul Connect integration followed that pattern exactly, as seen in #7396. However, on reviewing the PR we realized having the `expose` flag on the proxy stanza inseperably ties together the automatic path generation with every HTTP/gRPC defined on the service. This makes sense in Consul's context, because a service definition is reasonably associated with a single "task". With Nomad's group level service definitions however, there is a reasonable expectation that a service definition is more abstractly representative of multiple services within the task group. In this case, one would want to define checks of that service which concretely make HTTP or gRPC requests to different underlying tasks. Such a model is not possible with the course `proxy.expose` flag. Instead, we now have the flag made available within the check definitions themselves. By making the expose feature resolute to each check, it is possible to have some HTTP/gRPC checks which make use of the envoy exposed paths, as well as some HTTP/gRPC checks which make use of some orthongonal port-mapping to do checks on some other task (or even some other bound port of the same task) within the task group. Given this example, group "server-group" { network { mode = "bridge" port "forchecks" { to = -1 } } service { name = "myserver" port = 2000 connect { sidecar_service { } } check { name = "mycheck-myserver" type = "http" port = "forchecks" interval = "3s" timeout = "2s" method = "GET" path = "/classic/responder/health" expose = true } } } Nomad will automatically inject (via job endpoint mutator) the extrapolated expose path configuration, i.e. expose { path { path = "/classic/responder/health" protocol = "http" local_path_port = 2000 listener_port = "forchecks" } } Documentation is coming in #7440 (needs updating, doing next) Modifications to the `countdash` examples in hashicorp/demo-consul-101#6 which will make the examples in the documentation actually runnable. Will add some e2e tests based on the above when it becomes available.

shoenig · 2020-03-26T21:40:29Z

Closing in favor of #7515

…hecks Part of #6120 Building on the support for enabling connect proxy paths in #7323, this change adds the ability to configure the 'service.check.expose' flag on group-level service check definitions for services that are connect-enabled. This is a slight deviation from the "magic" that Consul provides. With Consul, the 'expose' flag exists on the connect.proxy stanza, which will then auto-generate expose paths for every HTTP and gRPC service check associated with that connect-enabled service. A first attempt at providing similar magic for Nomad's Consul Connect integration followed that pattern exactly, as seen in #7396. However, on reviewing the PR we realized having the `expose` flag on the proxy stanza inseperably ties together the automatic path generation with every HTTP/gRPC defined on the service. This makes sense in Consul's context, because a service definition is reasonably associated with a single "task". With Nomad's group level service definitions however, there is a reasonable expectation that a service definition is more abstractly representative of multiple services within the task group. In this case, one would want to define checks of that service which concretely make HTTP or gRPC requests to different underlying tasks. Such a model is not possible with the course `proxy.expose` flag. Instead, we now have the flag made available within the check definitions themselves. By making the expose feature resolute to each check, it is possible to have some HTTP/gRPC checks which make use of the envoy exposed paths, as well as some HTTP/gRPC checks which make use of some orthongonal port-mapping to do checks on some other task (or even some other bound port of the same task) within the task group. Given this example, group "server-group" { network { mode = "bridge" port "forchecks" { to = -1 } } service { name = "myserver" port = 2000 connect { sidecar_service { } } check { name = "mycheck-myserver" type = "http" port = "forchecks" interval = "3s" timeout = "2s" method = "GET" path = "/classic/responder/health" expose = true } } } Nomad will automatically inject (via job endpoint mutator) the extrapolated expose path configuration, i.e. expose { path { path = "/classic/responder/health" protocol = "http" local_path_port = 2000 listener_port = "forchecks" } } Documentation is coming in #7440 (needs updating, doing next) Modifications to the `countdash` examples in hashicorp/demo-consul-101#6 which will make the examples in the documentation actually runnable. Will add some e2e tests based on the above when it becomes available.

github-actions · 2022-11-19T02:28:09Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

shoenig mentioned this pull request Mar 20, 2020

cli: Fix sample connect check example #7389

Closed

shoenig requested review from nickethier and schmichael March 20, 2020 15:41

shoenig added 3 commits March 23, 2020 15:08

client: use consistent name for struct receiver parameter

a5a3dce

This helps reduce the number of squiggly lines in Goland.

jobspec: parse multi expose.path instead of explicit slice

5c5a3ab

shoenig force-pushed the f-connect-expose-checks branch from 8e1972f to 5c5a3ab Compare March 23, 2020 21:08

shoenig added 2 commits March 23, 2020 15:08

connect: fixup some leftover debugging comments

9da0de2

shoenig force-pushed the f-connect-expose-checks-auto branch from 862466b to 9da0de2 Compare March 23, 2020 21:08

pruett deployed to Netlify Preview March 23, 2020 21:15 View deployment

shoenig mentioned this pull request Mar 23, 2020

docs: add Envoy expose path configurability to changelog #7441

Closed

nickethier reviewed Mar 24, 2020

View reviewed changes

shoenig force-pushed the f-connect-expose-checks branch from 5c5a3ab to 42cb142 Compare March 24, 2020 19:23

shoenig mentioned this pull request Mar 26, 2020

connect: enable automatic expose paths for TG service checks #7515

Merged

shoenig closed this Mar 26, 2020

shoenig deleted the f-connect-expose-checks-auto branch June 15, 2021 14:29

github-actions bot locked as resolved and limited conversation to collaborators Nov 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

connect: enable configuring proxy.expose.checks easy button #7396

connect: enable configuring proxy.expose.checks easy button #7396

shoenig commented Mar 20, 2020 •

edited

Loading

nickethier left a comment

nickethier Mar 24, 2020

shoenig Mar 24, 2020

nickethier Mar 24, 2020

shoenig Mar 24, 2020

shoenig commented Mar 24, 2020 •

edited

Loading

shoenig commented Mar 26, 2020

github-actions bot commented Nov 19, 2022

connect: enable configuring proxy.expose.checks easy button #7396

connect: enable configuring proxy.expose.checks easy button #7396

Conversation

shoenig commented Mar 20, 2020 • edited Loading

nickethier left a comment

Choose a reason for hiding this comment

nickethier Mar 24, 2020

Choose a reason for hiding this comment

shoenig Mar 24, 2020

Choose a reason for hiding this comment

nickethier Mar 24, 2020

Choose a reason for hiding this comment

shoenig Mar 24, 2020

Choose a reason for hiding this comment

shoenig commented Mar 24, 2020 • edited Loading

shoenig commented Mar 26, 2020

github-actions bot commented Nov 19, 2022

shoenig commented Mar 20, 2020 •

edited

Loading

shoenig commented Mar 24, 2020 •

edited

Loading