consul/connect: fix regression where client connect images ignored

[shoenig]

Nomad v1.0.0 introduced a regression where the client configurations
for connect.sidecar_image and connect.gateway_image would be
ignored despite being set. This PR restores that functionality.

There was a missing layer of interpolation that needs to occur for
these parameters. Since Nomad 1.0 now supports dynamic envoy versioning
through the ${NOMAD_envoy_version} psuedo variable, we basically need
to first interpolate

  ${connect.sidecar_image} => envoyproxy/envoy:v${NOMAD_envoy_version}

then use Consul at runtime to resolve to a real image, e.g.

  envoyproxy/envoy:v${NOMAD_envoy_version} => envoyproxy/envoy:v1.16.0

Of course, if the version of Consul is too old to provide an envoy
version preference, we then need to know to fallback to the old
version of envoy that we used before.

  envoyproxy/envoy:v${NOMAD_envoy_version} => envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09

Beyond that, we also need to continue to support jobs that set the
sidecar task themselves, e.g.

  sidecar_task { config { image: "custom/envoy" } }

which itself could include the pseudo envoy version variable.

Fixes #9618

[shoenig]

Spot check every combination.

From lowest to highest precedence:
consul version: consul_old (envoy v1.11.2), consul_new (envoy v1.16.0)
client.meta: meta_unset, meta_set (envoy v1.14.4)
sidecar_task.image: image_unset, image_set (envoy v1.15.3)

Consul      meta.       image.         Expect    OK
----------------------------------------------------
consul_old  meta_unset  image_unset -> v1.11.2   [y]
consul_old  meta_unset  image_set   -> v1.15.3   [y]
consul_old  meta_set    image_unset -> v1.14.4   [y]
consul_old  meta_set    image_set   -> v1.15.3   [y]
consul_new  meta_unset  image_unset -> v1.16.0   [y]
consul_new  meta_unset  image_set   -> v1.15.3   [y]
consul_new  meta_set    image_unset -> v1.14.4   [y]
consul_new  meta_set    image_set   -> v1.15.3   [y]

[tgross]

LGTM

[tgross]

Solid test coverage here 👍

[HarryHarcourt]

We are running Nomad v1.0.2
And either this does not work, or we are trying to set the value in the wrong place.
Any help would be appreciated, we are putting this in our client agent configuration:
options {
meta.connect.sidecar_image = "artifactory.xxxx.com/docker/envoy:1.11.2"
}
But do not see any change when running nomad node status -self -verbose
Meta
connect.gateway_image = envoyproxy/envoy:v${NOMAD_envoy_version}
connect.log_level = info
connect.proxy_concurrency = 1
connect.sidecar_image = envoyproxy/envoy:v${NOMAD_envoy_version}

[tgross]

Hi @HarryHarcourt! I think you want:

client {
  meta {
    "connect.sidecar_image" = "artifactory.standard.com/docker/envoy:1.11.2"
  }
}

But if that doesn't work for you please open a new issue. By the way, the options block is deprecated since Nomad 0.9 and IIRC it doesn't get the new configuration fields.

[HarryHarcourt]

Confirmed that this works.
If this is moving to an options approach, what plugin would this be configured against? There is no documentation to support this approach.

[idrennanvmware]

@HarryHarcourt its just the standard nomad "client" configuration. Not specific plugins or drivers around them.

{
    "client": {
        "enabled": true,
        "network_interface" : "???",
        "meta": ["connect.sidecar_image": "{{ default_docker_registry }}/{{ images_dict_data.envoy }}", "connect.gateway_image": "{{ default_docker_registry }}/{{ images_dict_data.envoy }}"
        ]
    }
}

Please ignore our templating lol

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.