Allow control plane components to specify concurrency

[mateiidavid]

Concurrency in Linkerd is configured globally through a proxy.cores value or it can be configured through the injector. Control plane components cannot be configured through the injector since they come with the proxy pre-configured. In certain cases, it is necessary for proxies to specify their concurrency. Even though a global value exists, that directly influences all workloads in the cluster which can have other implications.

Each control plane component has a <component>ProxyResources field that is used to configure limits and requests. We overload this field and allow cores to be specified. We do not expect this value to be used often since directly specifying the number of cores might not make a lot of sense -- this does offer an escape hatch when the control plane components can't scale up to meet bursty workloads.

Note for reviewers

Click me for dig function docs

Overloading the operator with an additional field does not lead to failures. I have a couple of scenarios I tested and instructions on how to test locally.

To test:

1. bin/docker-build
2. Create a k3d cluster and import the images, bin/image-import --k3d --cluster <name>
3. Install through Helm (or CLI)

$ helm install linkerd \
       --namespace linkerd \
       --set destinationProxyResources.cores="2" \
       --set destinationProxyResources.memory.request="250Mi" \
       --set-file identityTrustAnchorsPEM=ca.crt \
       --set-file identity.issuer.tls.crtPEM=issuer.crt \
       --set-file identity.issuer.tls.keyPEM=issuer.key \
       --set linkerdVersion=(bin/root-tag) \
       charts/linkerd-control-plane/

I tested the following scenarios:

(1) Per-component default is applied

 $ helm install linkerd \
          --namespace linkerd \
          --set destinationProxyResources.cores="2" \
          --set destinationProxyResources.cpu.limit="3" \
          --set destinationProxyResources.memory.request="250Mi" \
          --set-file identityTrustAnchorsPEM=ca.crt \
          --set-file identity.issuer.tls.crtPEM=issuer.crt \
          --set-file identity.issuer.tls.keyPEM=issuer.key \
          --set linkerdVersion=(bin/root-tag) \
          charts/linkerd-control-plane/

$ k get po -n linkerd linkerd-destination-fc4bcdfb7-g7hfg -o yaml | yq '.spec.containers[] | select ( .name == "linkerd-proxy")' | rg 'CORES' -A 1
  - name: LINKERD2_PROXY_CORES
    value: "2"

$ k get po -n linkerd linkerd-destination-fc4bcdfb7-g7hfg -o yaml | yq '.spec.containers[] | select ( .name == "linkerd-proxy").resources'
limits:
  cpu: "3"
requests:
  cpu: "3"
  memory: 250Mi

(2) When nothing is specified (globally or per-component) we roll with the default.

$ helm install linkerd \
       --namespace linkerd \
       --set-file identityTrustAnchorsPEM=ca.crt \
       --set-file identity.issuer.tls.crtPEM=issuer.crt \
       --set-file identity.issuer.tls.keyPEM=issuer.key \
       --set linkerdVersion=(bin/root-tag) \
       charts/linkerd-control-plane/

$ k get po -n linkerd linkerd-destination-7bb9687899-8k7gf -o yaml |  yq '.spec.containers[] | select ( .name == "linkerd-proxy")' | rg 'CORES' -A 1
<nothing>

(3) When choosing between global and per-component, per-component is the override

$ helm install linkerd \
       --namespace linkerd \
       --set destinationProxyResources.cores="2" \
       --set proxy.cores="3" \
       --set destinationProxyResources.memory.request="250Mi" \
       --set-file identityTrustAnchorsPEM=ca.crt \
       --set-file identity.issuer.tls.crtPEM=issuer.crt \
       --set-file identity.issuer.tls.keyPEM=issuer.key \
       --set linkerdVersion=(bin/root-tag) \
       charts/linkerd-control-plane/

$  k get po -n linkerd linkerd-destination-6d7f86fd48-v7zkz -o yaml | yq '.spec.containers[] | select ( .name == "linkerd-proxy")' | rg 'CORES' -A 1
  - name: LINKERD2_PROXY_CORES
    value: "2"

(4) Specifying only a global means components respect the global

$ helm install linkerd \
       --namespace linkerd \
       --set proxy.cores="3" \
       --set destinationProxyResources.memory.request="250Mi" \
       --set-file identityTrustAnchorsPEM=ca.crt \
       --set-file identity.issuer.tls.crtPEM=issuer.crt \
       --set-file identity.issuer.tls.keyPEM=issuer.key \
       --set linkerdVersion=(bin/root-tag) \
       charts/linkerd-control-plane/

$ k get po -n linkerd linkerd-destination-56c786ff49-kl6r7 -o yaml | yq '.spec.containers[] | select ( .name == "linkerd-proxy")' | rg 'CORES' -A 1
  - name: LINKERD2_PROXY_CORES
    value: "3"

(5) We can upgrade and specify a per-component value to override the global

 $ helm upgrade linkerd \
       --namespace linkerd \
       --set destinationProxyResources.cores="1" \
       --set destinationProxyResources.memory.request="250Mi" \
       --set-file identityTrustAnchorsPEM=ca.crt \
       --set-file identity.issuer.tls.crtPEM=issuer.crt \
       --set-file identity.issuer.tls.keyPEM=issuer.key \
       --set linkerdVersion=(bin/root-tag) \
       charts/linkerd-control-plane/

$ k get po -n linkerd linkerd-destination-74968c6999-q6kwn -o yaml |  yq '.spec.containers[] | select ( .name == "linkerd-proxy")' | rg 'CORES' -A 1
  - name: LINKERD2_PROXY_CORES
    value: "1"

The tests show some of the basic install / upgrade flows that people would follow. Since the value will override only when specified (and is not documented) it should not affect existing installations unless users explicitly opt-in.

[alpeb]

LGTM. The <component>ProxyResources docs only make reference to "proxy.resources for sub-fields". Do we want to leave this new option undocumented?

[mateiidavid]

Yeah, I think so. Since this might be subject to change, we should maybe document it only when it's polished.