local_service_port does not work with hcl job format

[davejhilton]

Nomad version

0.10.4 (I've also tried this on 0.10.3 - same results)

Operating system and Environment details

CentOS 7.7 - though I suspect the issue is OS-independent

Issue

When submitting jobs through the cli with the hcl job format, the local_service_port and local_service_address properties (added in #6358) are not respected. These properties only appear to be respected if jobs are submitted as JSON through the http api.

Reproduction steps

1. Run the job below via the CLI on any basic nomad cluster: nomad job run countdash.hcl
2. Run nomad job inspect countdash > countdash.json
3. cat countdash.json, and see that the LocalServicePort property has a value of "0", instead of the expected "9001".
4. Modify that json file—in the count-api's service block, change it to say "LocalServicePort": "9001". (Due to Changes to connect stanza get ignored on job re-run  #6459 you will also have to edit another property for it to take effect—change the value of the "FOO" variable in the env block from "BAR", to "FOOBAR")
5. Submit the modified job via the http api, with curl: curl -XPOST -H "Content-Type: application/json" -d @countdash.json http://127.0.0.1:4646/v1/jobs
6. Observe that the local service port is now correctly reflected when inspecting the job (nomad job inspect | grep "LocalServicePort")
7. Modify your ORIGINAL job file (the HCL version) and change the local_service_port to something else, like 10001
8. Re-run the job with the cli: nomad job run countdash.hcl
9. Observe that once again, inspecting the job shows LocalServicePort has been reverted to "0"

Job file

Below is the "countdash" example job from the nomad consul-connect integration guide... except that I modified it slightly to show this issue. The only changes I made were:

• adding the port "api_http" block to the "api" group, with to = 9001
• changing the count-api service's port from 9001 to the port label "api_http"
• adding the proxy block to the connect sidecar, with the local_service_port = 9001 option
• adding an (irrelevant) env block to the "web" task, to make the reproduction steps above easier

This minimally reproduce my current setup, which is a hybrid of consul-connect-enabled and non-connect services, as part of a larger transition plan.

job "countdash" {
  datacenters = ["dc1"]

  group "api" {
    network {
      mode = "bridge"

      port "api_http" {
        to = 9001
      }
    }

    service {
      name = "count-api"
      port = "api_http"

      connect {
        sidecar_service {
          proxy {
            local_service_port = 9001
          }
        }
      }
    }

    task "web" {
      driver = "docker"

      env {
        FOO = "BAR"
      }

      config {
        image = "hashicorpnomad/counter-api:v1"
      }
    }
  }

  group "dashboard" {
    network {
      mode = "bridge"

      port "http" {
        static = 9002
        to     = 9002
      }
    }

    service {
      name = "count-dashboard"
      port = "9002"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "count-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "dashboard" {
      driver = "docker"

      env {
        COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
      }

      config {
        image = "hashicorpnomad/counter-dashboard:v1"
      }
    }
  }
}

[shoenig]

Thank you for reporting this, @davejhilton !
There were a couple of bugs in here, which should be fixed for v0.11.1

• two fixes for inspect on connect proxy #7690
• connect block blank in job inspect output #7397

Definitely re-open or file new tickets if something still does not work!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.