Ability to configure Envoy concurrency/worker thread count for Nomad jobs [feature request]

[chrisboulton]

Feature Request

By default, Envoy configures itself with as many worker threads as there are CPU cores available (per https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-concurrency)

In some environments, this may be okay but similarly to how resources (CPU and memory) are assignable to Envoy, it is desirable to be able to tune this -- specifically in our environment where we're going to be running a large number of sidecar containers and a large number of threads leads to increased memory utilisation, and in the case of smaller throughput services lower connection pool hit ratios. Lyft themselves suggest that in their environment - they tune Envoy sidecars down to a small number of threads as is reasonable for the proxied service.

This is (hopefully) possible today, by overriding sidecar_task.config.args to supply a custom --concurrency option, kinda like so:

args = [
  "--concurrency", "${NOMAD_envoy_threads}",
  "-c", "${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
  "-l", ${meta.connect.log_level}",
  "--disable-hot-restart",
]

It'd be nice to not have to do that and possibly end up out of sync with how Nomad is configuring Envoy.

Possible Implementation

In my mind, this is something that should be tuneable per sidecar task, so either a parameter (possibly a known metadata key/value like) should work here, or alternatively an environment variable that can be set on the task. The value should possibly default to 0 (the Envoy default):

[shoenig]

Thanks for pointing this out, @chrisboulton !

Something like ${meta.connect.proxy_concurrency} would be consistent with existing meta fields for configuring the connect proxy task, and would allow configuring the value at the job, group, or task level. I actually think we should set the default to 1 going forward (last minute backwards incompatibility before 1.0), as doing so would be more reasonable for the default amount of cpu resources assigned to the sidecar (cpu = 250 # mhz)

[chrisboulton]

@shoenig: Yup, your proposed implementation sounds great. It sounds like it'd give us control/defaults at the client level as well similar to how custom Envoy images, the log level etc is handled. The proposed default value of 1 makes sense also (we ended up defaulting to 2 in our environment)

As an anecdote, we implemented the workaround I suggested, and it really helped us stabilise the memory usage of Envoy (especially at startup), in our rather large deployment.

It'd be wonderful if y'all could bring this down before 1.0 is cut, and we'd be happy to test as always. 🥳

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.