Using uppercase characters in the service name causes the envoy proxy to fail

[rbjorklin]

Nomad version

Nomad v0.11.0-beta1 (a7a7d1284f758e8c431ad4f251131f514da0066d)

Operating system and Environment details

# cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)

Issue

Creating a task with a camelCase name causes envoy proxy to fail. If this is not supposed to work then please fix #5301 since both camelCase and snake_case are currently broken.

Reproduction steps

Run the attached job file.

Job file (if appropriate)

graylog.txt

Nomad Client logs (if appropriate)

Output from nomad alloc status:
Task hook failed envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exit status 1

[shoenig]

Thanks you for taking the time to report this, @rbjorklin !

In working on reproducing this, I've found the behavior is coming from Consul rather than Nomad. I'm not sure yet if this is expected or if Nomad is invoking the bootstrap command incorrectly.

A small reproducible example:

# myApp.hcl 
service {
  name = "myApp"
  port = 9999
  connect { sidecar_service {} }
}

consul agent -dev -config-file=myApp.hcl

$ consul connect envoy -sidecar-for myApp -bootstrap
==> No sidecar proxy registered for myApp

However, if we just rename myApp to myapp, ...

# myapp.hcl 
service {
  name = "myapp"
  port = 9999
  connect { sidecar_service {} }
}

consul agent -dev -config-file=myapp.hcl

$ consul connect envoy -sidecar-for myapp -bootstrap
{
  "admin": {
    "access_log_path": "/dev/null",
    "address": {
      "socket_address": {
        "address": "127.0.0.1",
        "port_value": 19000
      }
    }
  },
  "node": {
    "cluster": "myapp",
    "id": "myapp-sidecar-proxy",
    "metadata": {
      "namespace": "default",
      "envoy_version": "1.13.0"
    }
  },
... (properly generated config) ...

The Consul -sidecar-for documentation does make a note about specifying Name vs ID,

The ID (not name if they differ) of the service instance this proxy will represent.

However in the capitalized case they still appear to be the same.

$ curl -s localhost:8500/v1/catalog/service/myApp-sidecar-proxy | jq -r '.[].ServiceID,.[].ServiceName'
myApp-sidecar-proxy
myApp-sidecar-proxy
$ curl -s localhost:8500/v1/catalog/service/myApp | jq -r '.[].ServiceID,.[].ServiceName'
myApp
myApp

[shoenig]

Looks like this is indeed a Consul issue hashicorp/consul#6765

[stale]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[shoenig]

Seems like we could at least add some job endpoint validation with a nice error message. Would be much less confusing than trying to debug what currently happens.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.