Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

consul/connect: default envoy concurrency to 1 #9487

Merged
merged 1 commit into from
Dec 1, 2020
Merged

consul/connect: default envoy concurrency to 1 #9487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ IMPROVEMENTS:
* consul: Support Consul namespace (Consul Enterprise) in client configuration. [[GH-8849](https://github.com/hashicorp/nomad/pull/8849)]
* consul/connect: Dynamically select envoy sidecar at runtime [[GH-8945](https://github.com/hashicorp/nomad/pull/8945)]
* consul/connect: Enable setting `datacenter` field on connect upstreams [[GH-8964](https://github.com/hashicorp/nomad/issues/8964)]
* consul/connect: Envoy concurrency now defaults to 1 rather than number of cores [[GH-9341](https://github.com/hashicorp/nomad/issues/9341)]
* csi: Support `nomad volume detach` with previously garbage-collected nodes. [[GH-9057](https://github.com/hashicorp/nomad/issues/9057)]
* csi: Relaxed validation requirements when checking volume capabilities with controller plugins, to accommodate existing plugin behaviors. [[GH-9049](https://github.com/hashicorp/nomad/issues/9049)]
* driver/docker: Upgrade pause container and detect architecture [[GH-8957](https://github.com/hashicorp/nomad/pull/8957)]
Expand Down
20 changes: 16 additions & 4 deletions client/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,17 +98,26 @@ const (
allocSyncRetryIntv = 5 * time.Second

// defaultConnectSidecarImage is the image set in the node meta by default
// to be used by Consul Connect sidecar tasks
// Update sidecar_task.html when updating this.
// to be used by Consul Connect sidecar tasks. As of Nomad 1.0, this value
// is only used as a fallback when the version of Consul does not yet support
// dynamic envoy versions.
defaultConnectSidecarImage = "envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09"

// defaultConnectGatewayImage is the image set in the node meta by default
// to be used by Consul Connect Gateway tasks.
// to be used by Consul Connect Gateway tasks. As of Nomad 1.0, this value
// is only used as a fallback when the version of Consul does not yet support
// dynamic envoy versions.
defaultConnectGatewayImage = defaultConnectSidecarImage

// defaultConnectLogLevel is the log level set in the node meta by default
// to be used by Consul Connect sidecar tasks
// to be used by Consul Connect sidecar tasks.
defaultConnectLogLevel = "info"

// defaultConnectProxyConcurrency is the default number of worker threads the
// connect sidecar should be configured to use.
//
// https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-concurrency
defaultConnectProxyConcurrency = "1"
)

var (
Expand Down Expand Up @@ -1403,6 +1412,9 @@ func (c *Client) setupNode() error {
if _, ok := node.Meta["connect.log_level"]; !ok {
node.Meta["connect.log_level"] = defaultConnectLogLevel
}
if _, ok := node.Meta["connect.proxy_concurrency"]; !ok {
node.Meta["connect.proxy_concurrency"] = defaultConnectProxyConcurrency
}

return nil
}
Expand Down
1 change: 1 addition & 0 deletions nomad/job_endpoint_hook_connect.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ var (
"args": []interface{}{
"-c", structs.EnvoyBootstrapPath,
"-l", "${meta.connect.log_level}",
"--concurrency", "${meta.connect.proxy_concurrency}",
"--disable-hot-restart",
},
}
Expand Down
62 changes: 34 additions & 28 deletions website/pages/docs/job-specification/sidecar_task.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,49 +61,54 @@ group service has a [`sidecar_service`][sidecar_service] stanza.
The default sidecar task is equivalent to:

```hcl
sidecar_task {
name = "connect-proxy-<service>"
sidecar_task {
name = "connect-proxy-<service>"

lifecycle {
hook = "prestart"
sidecar = true
}
lifecycle {
hook = "prestart"
sidecar = true
}

driver = "docker"
config {
image = "${meta.connect.sidecar_image}"
args = [
"-c",
"${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
"-l",
"${meta.connect.log_level}"
]
}
driver = "docker"
config {
image = "${meta.connect.sidecar_image}"
args = [
"-c",
"${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
"-l",
"${meta.connect.log_level}",
"--concurrency",
"${meta.connect.proxy_concurrency}",
"--disable-hot-restart"
]
}

logs {
max_files = 2
max_file_size = 2 # MB
}
logs {
max_files = 2
max_file_size = 2 # MB
}

resources {
cpu = 250 # MHz
memory = 128 # MB
}
resources {
cpu = 250 # MHz
memory = 128 # MB
}

shutdown_delay = "5s"
}
shutdown_delay = "5s"
}
```

The `meta.connect.sidecar_image` and `meta.connect.log_level` are [_client_
The `meta.connect.sidecar_image`, `meta.connect.log_level`, and
`meta.connect.proxy_concurrency` variables are [_client_
configurable][nodemeta] variables with the following defaults:

- `sidecar_image` - `(string: "envoyproxy/envoy:v${NOMAD_envoy_version}")` - The official
upstream Envoy Docker image, where `${NOMAD_envoy_version}` is resolved automatically
by a query to Consul.
- `log_level` - `(string: "info")` - Envoy sidecar log level. "`debug`" is useful for
debugging Connect related issues.
- `proxy_concurrency` - `(string: "1")` - The number of [worker threads][worker_threads] the Envoy
sidecar will run.

`meta.connect.sidecar_image` can be configured at the job, group, or task level.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this line for now because it's not actually true. (Would like for it to be, though! #7221)

Custom images can make use of Consul's preferred Envoy version by making use of
Nomad's version interpolation, e.g.

Expand Down Expand Up @@ -169,3 +174,4 @@ The following example configures resources for the sidecar task and other config
[resources]: /docs/job-specification/resources 'Nomad resources Job Specification'
[logs]: /docs/job-specification/logs 'Nomad logs Job Specification'
[nodemeta]: /docs/configuration/client#meta
[worker_threads]: https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-concurrency
17 changes: 14 additions & 3 deletions website/pages/docs/upgrade/upgrade-specific.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ standard upgrade flow.

### HCL2 for Job specification

Nomad 1.0.0 adopts HCL2 for parsing the job spec. HCL2 extends HCL with more
Nomad v1.0.0 adopts HCL2 for parsing the job spec. HCL2 extends HCL with more
expression and reuse support, but adds some stricter schema for HCL blocks (a.k.a. stanzas). Check [HCL](/docs/job-specification/hcl2) for more details.

### Signal used when stopping Docker tasks
Expand Down Expand Up @@ -73,12 +73,12 @@ Nomad. The specific configuration values replaced are:

### Envoy proxy versions

Nomad 1.0.0 changes the behavior around the selection of Envoy version used
Nomad v1.0.0 changes the behavior around the selection of Envoy version used
for Connect sidecar proxies. Previously, Nomad always defaulted to Envoy v1.11.2
if neither the `meta.connect.sidecar_image` parameter or `sidecar_task` stanza
were explicitly configured. Likewise the same version of Envoy would be used for
Connect ingress gateways if `meta.connect.gateway_image` was unset. Starting with
Nomad 1.0.0, each Nomad Client will query Consul for a list of supported Envoy
Nomad v1.0.0, each Nomad Client will query Consul for a list of supported Envoy
versions. Nomad will make use of the latest version of Envoy supported by the
Consul agent when launching Envoy as a Connect sidecar proxy. If the version of
the Consul agent is older than v1.7.8, v1.8.4, or v1.9.0, Nomad will fallback to
Expand All @@ -94,6 +94,15 @@ the time of the upgrade for each node will ensure Connect workloads are properly
rescheduled onto nodes in such a way that the Nomad Clients, Consul agents, and
Envoy sidecar tasks maintain compatibility with one another.

### Envoy worker threads

Nomad v1.0.0 changes the default behaviour around the number of worker threads
created by the Envoy sidecar proxy when using Consul Connect. Previously, the
Envoy [`--concurrency`][envoy_concurrency] argument was left unset, which caused
Envoy to spawn as many worker threads as logical cores available on the CPU. The
`--concurrency` value now defaults to `1` and can be configured by setting the
[`meta.connect.proxy_concurrency`][proxy_concurrency] property in client configuration.

## Nomad 0.12.8

### Docker volume mounts
Expand Down Expand Up @@ -859,6 +868,7 @@ deleted and then Nomad 0.3.0 can be launched.
[drain-api]: /api-docs/nodes#drain-node
[drain-cli]: /docs/commands/node/drain
[dst]: /docs/job-specification/periodic#daylight-saving-time
[envoy_concurrency]: https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-concurrency
[gh-6787]: https://github.com/hashicorp/nomad/issues/6787
[gh-8457]: https://github.com/hashicorp/nomad/issues/8457
[gh-9148]: https://github.com/hashicorp/nomad/issues/9148
Expand All @@ -870,6 +880,7 @@ deleted and then Nomad 0.3.0 can be launched.
[plugins]: /docs/drivers/external
[preemption-api]: /api-docs/operator#update-scheduler-configuration
[preemption]: /docs/internals/scheduling/preemption
[proxy_concurrency]: /docs/job-specification/sidecar_task#proxy_concurrency
[reserved]: /docs/configuration/client#reserved-parameters
[task-config]: /docs/job-specification/task#config
[tls-guide]: https://learn.hashicorp.com/tutorials/nomad/security-enable-tls
Expand Down