Custom Connect Sidecar Checks #10524

notnoop · 2021-05-06T14:56:56Z

This is WIP PR to ensure I'm in the right path before committing to this approach. This is all very foreign to me.

This PR adds supports for custom checks for connect's sidecar_service, in a similar fashion to normal services. When specified, it overrides the hardcoded "Connect Sidecar Listening" TCP check inserted by Nomad.

A sample connect stanza might be:

connect {
  sidecar_service {
    tags = ["mytest"]
    proxy {
      upstreams {
        destination_name = "count-api"
        local_bind_port  = 8080
      }
    }

    check {
      type     = "http"
      path     = "/"
      interval = "10s"
      timeout  = "3s"
    }
  }
}

I have few comments/questions that are inlined.

Related to #9773

notnoop · 2021-05-06T15:01:20Z

command/agent/consul/connect.go

-		Proxy:   proxy,
-		Checks: api.AgentServiceChecks{
+	var checks []*api.AgentServiceCheck
+	if len(css.Checks) == 0 {


With this change, there is no way to actually disable sidecar checks. The job author must specify a check or we insert the default TCP ones. This seems a bit reasonable.

This is also the only option if registering with Consul directly; this approach is just following suite

notnoop · 2021-05-06T15:03:41Z

nomad/structs/services.go

+		//if c.PortLabel != "" {
+		//	mErr.Errors = append(mErr.Errors, fmt.Errorf("Check %s invalid: port is automatically assigned", c.Name))
+		//	continue
+		//}


Not sure if we should allow setting port labels here; as the sidecar service doesn't specify ports.

notnoop · 2021-05-06T15:06:25Z

command/agent/consul/connect.go

+	} else {
+		checks = make([]*api.AgentServiceCheck, len(css.Checks))
+		for i, c := range css.Checks {
+			check, err := createCheckReg(serviceId, "", c, cMapping.HostIP, cMapping.Value, "")


Not sure if I'm doing this correctly, and what host/port should be used. When I tried, I can successfully set a custom TCP check; but cannot seem to get an HTTP to pass. The default envoyproxy seems to reject HTTP connection?!

I sort of imagined being able to create checks that somehow made use of NOMAD_ENVOY_ADMIN_ADDR_<service>, where envoy provides some default paths one might query https://www.envoyproxy.io/docs/envoy/latest/operations/admin to represent the envoy's health.

The only way that will work is if the Consul agent is able to make a connection to that ip:port, which is going to be 127.0.0.1:1900x inside the network namespace - which means Nomad would need to inject a new dynamic port mapping to that envoy admin port.

notnoop · 2021-05-06T16:56:42Z

command/agent/consul/connect.go

@@ -120,7 +117,25 @@ func connectSidecarRegistration(serviceId string, css *structs.ConsulSidecarServ
 				Name:         "Connect Sidecar Aliasing " + serviceId,
 				AliasService: serviceId,
 			},


Unclear what the value of this Aliasing service id - do we need to insert it even when checks are overriden?

The alias basically means "if you request health for service X, send the request to me instead" - which is useful for connect, where requests must go through the sidecar proxy. I think yes, this should always be there

shoenig

@notnoop this is definitely along the lines of the original suggestion - being able to specify custom checks on the sidecar service.

Taking into consideration the original ask of #9773 and the amount of complexity in enabling custom sidecar checks that actually do something, I now wonder if it would be better to just add a kind of disable_default_tcp_check option. Otherwise, to have envoy be able to respond to these custom checks, we're going to need to basically hack-up the expose machinery, making it possible for envoy creates an exposed listener for itself. Re-reading that issue, nobody is really asking for that - they just want the TCP checks to stop.

What are your thoughts? The simpler option is divergent from Consul, but the checks option is going to be more work.

Thinking out-loud, there might also be an in-between approach, where the sidecar_service.checks definition is a minimal subset of a normal check, where Nomad owns all of the fields except maybe path and tags - making so that we just need to automatically manage an injected port mapping and the expose listener.

shoenig · 2021-05-06T16:27:34Z

command/agent/consul/connect.go

-		Proxy:   proxy,
-		Checks: api.AgentServiceChecks{
+	var checks []*api.AgentServiceCheck
+	if len(css.Checks) == 0 {


This is also the only option if registering with Consul directly; this approach is just following suite

shoenig · 2021-05-06T16:40:40Z

command/agent/consul/connect.go

+	} else {
+		checks = make([]*api.AgentServiceCheck, len(css.Checks))
+		for i, c := range css.Checks {
+			check, err := createCheckReg(serviceId, "", c, cMapping.HostIP, cMapping.Value, "")


I sort of imagined being able to create checks that somehow made use of NOMAD_ENVOY_ADMIN_ADDR_<service>, where envoy provides some default paths one might query https://www.envoyproxy.io/docs/envoy/latest/operations/admin to represent the envoy's health.

The only way that will work is if the Consul agent is able to make a connection to that ip:port, which is going to be 127.0.0.1:1900x inside the network namespace - which means Nomad would need to inject a new dynamic port mapping to that envoy admin port.

notnoop · 2021-05-06T17:24:12Z

now wonder if it would be better to just add a kind of disable_default_tcp_check option

That sounds reasonable given my experience too, indeed - it's kinda hard to specify a check that targets the envoy directly without it being actually for the backend service. I'll have another PR ready this afternoon.

notnoop · 2021-05-10T14:40:34Z

Superseded by #10531 . Can resurrect to handle the extra complexity if the demand is there.

github-actions · 2022-11-23T02:23:48Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Mahmood Ali added 4 commits May 6, 2021 10:49

Prepare structs for sidecar service checks

2a4adab

http: Perform api->structs conversion

5d5e7ae

Propagate custom services to consul

d195c34

Update hcl1 parser just because

0ed745b

notnoop added this to the 1.1.0 milestone May 6, 2021

notnoop self-assigned this May 6, 2021

notnoop commented May 6, 2021

View reviewed changes

notnoop requested a review from shoenig May 6, 2021 15:07

Mahmood Ali added 3 commits May 6, 2021 12:14

fixup! http: Perform api->structs conversion

5fdb909

fixup! http: Perform api->structs conversion

1d1ebf0

fixup! Update hcl1 parser just because

96248da

notnoop commented May 6, 2021

View reviewed changes

fixup! Propagate custom services to consul

dcd2533

vercel bot deployed to Preview – nomad-storybook-and-ui May 6, 2021 16:58 View deployment

vercel bot temporarily deployed to Preview – nomad May 6, 2021 16:58 Inactive

shoenig reviewed May 6, 2021

View reviewed changes

notnoop closed this May 6, 2021

notnoop reopened this May 6, 2021

notnoop mentioned this pull request May 6, 2021

Support disabling TCP checks for connect sidecar services #10531

Merged

notnoop closed this May 10, 2021

github-actions bot locked as resolved and limited conversation to collaborators Nov 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom Connect Sidecar Checks #10524

Custom Connect Sidecar Checks #10524

notnoop commented May 6, 2021 •

edited

Loading

notnoop May 6, 2021

shoenig May 6, 2021

notnoop May 6, 2021

notnoop May 6, 2021

shoenig May 6, 2021

notnoop May 6, 2021

shoenig May 6, 2021

shoenig left a comment

shoenig May 6, 2021

shoenig May 6, 2021

notnoop commented May 6, 2021

notnoop commented May 10, 2021

github-actions bot commented Nov 23, 2022

Custom Connect Sidecar Checks #10524

Custom Connect Sidecar Checks #10524

Conversation

notnoop commented May 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shoenig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notnoop commented May 6, 2021

notnoop commented May 10, 2021

github-actions bot commented Nov 23, 2022

notnoop commented May 6, 2021 •

edited

Loading