services: enable setting arbitrary address value in service registrations #12720

shoenig · 2022-04-20T18:03:35Z

This PR introduces the address field in the service block so that Nomad
or Consul services can be registered with a custom .Address. to advertise.

The address can be an IP address or domain name. If the address field is
set, the service.address_mode must be set in auto mode.

Note this slightly changes the output of nomad service info, such that :0-ports
are no longer included in the CLI output.

Closes #12722
Closes #2770
Closes #4815
Closes #3629

Basic Example

job "basic" {
  datacenters = ["dc1"]

  group "basicN" {
    service {
      name     = "basic"
      address  = "basic.example.com"
      provider = "nomad"
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }

  group "basicC" {
    service {
      name     = "basic"
      address  = "basic.example.com"
      provider = "consul"
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }
}

➜ nomad service info basic 
Job ID  Address            Tags  Node ID   Alloc ID
basic   basic.example.com  []    a75b1a27  38b6867d

➜ curl -s localhost:8500/v1/catalog/service/basic | jq -r .[].ServiceAddress
basic.example.com

Interpolation Example

job "interp" {
  datacenters = ["dc1"]
  group "interpN" {
    service {
      name     = "interp"
      address  = "${attr.unique.network.ip-address}"
      port     = 9911
      provider = "nomad"
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }
  group "interpC" {
    service {
      name     = "interp"
      address  = "${attr.unique.network.ip-address}"
      port     = 9911
      provider = "consul"
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }
}

➜ nomad service info interp
Job ID  Address              Tags  Node ID   Alloc ID
interp  192.168.88.252:9911  []    7d3ce76e  7eb71559

➜ curl -s localhost:8500/v1/catalog/service/interp | jq -r '.[] | .ServiceAddress, .ServicePort'
192.168.88.252
9911

Example with Consul Checks

job "checks" {
  datacenters = ["dc1"]

  group "checksC" {
    service {
      name     = "checks"
      address  = "example.com"
      provider = "consul"
      port     = 80
      check {
        name     = "c1"
        path     = "/"
        type     = "http"
        interval = "3s"
        timeout  = "5s"
      }
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }
}

➜ curl -s localhost:8500/v1/agent/checks | jq -r '.[] | select(.Name=="c1")'
{
  "Node": "x52",
  "CheckID": "_nomad-check-f71cf56fdc9efb2fd924a76c83f89c37ea806c07",
  "Name": "c1",
  "Status": "passing",
  "Notes": "",
  "Output": "HTTP GET http://example.com:80/: 200 OK Output: <!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
  "ServiceID": "_nomad-task-fa6c1a17-a9a2-de2c-e237-17642d0a6018-group-checksC-checks-80",
  "ServiceName": "checks",
  "ServiceTags": [],
  "Type": "http",
  "Interval": "3s",
  "Timeout": "5s",
  "ExposedPort": 0,
  "Definition": {},
  "CreateIndex": 0,
  "ModifyIndex": 0
}

Bonus EC2 Public IPv4 Example

job "interp" {
  datacenters = ["dc1"]
  group "interpN" {
    service {
      name     = "interp"
      address  = "${attr.unique.platform.aws.public-ipv4}"
      port     = 9911
      provider = "nomad"
    }
    task "sleep" {
      driver = "exec"
      config {
        command = "sleep"
        args    = ["1000"]
      }
    }
  }
}

ubuntu@ip-172-31-15-70:~$ ./nomad service info interp
Job ID  Address           Tags  Node ID   Alloc ID
interp  3.80.11.106:9911  []    03b507af  403af357

schmichael

Looks great! Some options for docs updates and an ask for an interpolation test, but nothing that needs to block merging.

schmichael · 2022-04-21T21:09:47Z

api/services.go

-	//FIXME Id is unused. Remove?
-	Id                     string              `hcl:"id,optional"`


You answered that question! Thanks for getting rid of deadcode 👍

schmichael · 2022-04-21T21:15:32Z

client/serviceregistration/address.go

+// service or check registration. If no port label is specified (an empty value),
+// zero values are returned because no address could be resolved.


If no port label is specified (an empty value), zero values are returned because no address could be resolved.

I don't think this is quite accurate anymore? If no port label is specified address is returned which may or may not be empty.

Perhaps:

If no port label or an explicit address are specified, zero values are returned because no address could be resolved.

schmichael · 2022-04-21T21:19:26Z

client/serviceregistration/address.go

+			// A custom advertise address can be used with a port map; using the
+			// Value and ignoring the IP. The routing from your custom address to
+			// the group network address is DIY.
+			if mapping, exists := ports.Get(portLabel); exists {
+				return address, mapping.Value, nil
+			}


Took me a second to realize this is the "EC2" use case: advertise the public IP address with whatever port is defined.

Not sure it's worth expanding an already ample comment to include that though.

command/service_info.go

schmichael · 2022-04-21T23:21:15Z

website/content/docs/job-specification/service.mdx

@@ -118,7 +118,7 @@ Connect][connect] integration.
    If a `to` value is not set, the port falls back to using the allocated host port. The `port`
    field may be a numeric port or a port label specified in the same group's network stanza.

-  - `driver` - Advertise the port determined by the driver (e.g. Docker or rkt).
+  - `driver` - Advertise the port determined by the driver (e.g. Docker).


schmichael · 2022-04-21T23:47:23Z

website/content/docs/job-specification/service.mdx

+- `address` `(string: <optional>)` - Specifies a custom address to advertise in
+  Consul or Nomad service registration. Can be an IP address or domain name. If
+  set, `address_mode` must be in `auto` mode.


Let's note that interpolation is supported or perhaps even suggest using it. Perhaps:

address (string: <optional>) - Manually specifies a custom address to advertise. address_mode must be left on its default auto mode. This is most useful in combination with interpolation. For example to advertise the public IP address of an AWS EC2 node set this to ${unique.platform.aws.public-ipv4}

schmichael · 2022-04-21T23:51:26Z

client/taskenv/services.go

@@ -36,6 +36,7 @@ func InterpolateServices(taskEnv *TaskEnv, services []*structs.Service) []*struc

 		service.Name = taskEnv.ReplaceEnv(service.Name)
 		service.PortLabel = taskEnv.ReplaceEnv(service.PortLabel)
+		service.Address = taskEnv.ReplaceEnv(service.Address)


Let's add a test to assert this gets interpolated. We rarely touch this, so the risk of it getting accidentally broken is low, but given how critical interpolation is to this feature I think it's worth covering at least once.

jrasell · 2022-04-22T11:26:38Z

client/serviceregistration/address.go

-// address could be resolved.
+// GetAddress returns the IP (or custom advertise address) and port to use for a
+// service or check registration. If no port label is specified (an empty value),
+// zero values are returned because no address could be resolved.
 func GetAddress(


Nothing for this PR, but noting for the future that I wonder if it's worth changing this function signature to take a struct rather than a growing list of types.

This, and also I think the port mapping lookup and numeric interpolation fallback can be consolidated from now 4 places

jrasell

LGTM along with some of the comments already present. I really appreciate the updates to the comments particularly in the API.

A nice followup might be to relax the service name validation #11097.

…ions This PR introduces the `address` field in the `service` block so that Nomad or Consul services can be registered with a custom `.Address.` to advertise. The address can be an IP address or domain name. If the `address` field is set, the `service.address_mode` must be set in `auto` mode.

Co-authored-by: Michael Schurter <mschurter@hashicorp.com>

github-actions · 2022-10-15T02:46:10Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

vercel bot deployed to Preview – nomad-storybook-and-ui April 20, 2022 18:06 View deployment

shoenig force-pushed the f-arbitrary-addresses branch from f5c9d86 to ccc29d1 Compare April 20, 2022 19:47

vercel bot deployed to Preview – nomad-storybook-and-ui April 20, 2022 19:51 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 20, 2022 20:02 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 20, 2022 20:14 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 20, 2022 20:19 View deployment

shoenig force-pushed the f-arbitrary-addresses branch from ed41878 to d54d4ec Compare April 21, 2022 15:04

shoenig changed the title ~~wip seems ok~~ services: enable setting arbitrary address value in service registrations Apr 21, 2022

vercel bot deployed to Preview – nomad April 21, 2022 15:06 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 21, 2022 15:07 View deployment

shoenig force-pushed the f-arbitrary-addresses branch from d54d4ec to 84d059e Compare April 21, 2022 15:22

vercel bot deployed to Preview – nomad April 21, 2022 15:24 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 21, 2022 15:24 View deployment

shoenig force-pushed the f-arbitrary-addresses branch from 84d059e to 9ae5f1a Compare April 21, 2022 15:25

vercel bot deployed to Preview – nomad April 21, 2022 15:27 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 21, 2022 15:28 View deployment

shoenig force-pushed the f-arbitrary-addresses branch from 9ae5f1a to 96dacf2 Compare April 21, 2022 15:41

vercel bot deployed to Preview – nomad April 21, 2022 15:43 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 21, 2022 15:44 View deployment

shoenig added the theme/service-discovery label Apr 21, 2022

shoenig added this to the 1.3.0 milestone Apr 21, 2022

shoenig marked this pull request as ready for review April 21, 2022 16:27

shoenig requested review from jrasell and schmichael April 21, 2022 16:27

schmichael approved these changes Apr 21, 2022

View reviewed changes

jrasell reviewed Apr 22, 2022

View reviewed changes

jrasell approved these changes Apr 22, 2022

View reviewed changes

vercel bot deployed to Preview – nomad-storybook-and-ui April 22, 2022 13:17 View deployment

vercel bot deployed to Preview – nomad April 22, 2022 13:46 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 22, 2022 13:46 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui April 22, 2022 13:50 View deployment

shoenig and others added 4 commits April 22, 2022 09:14

services: format ipv6 in nomad service info output

2e26098

Co-authored-by: Michael Schurter <mschurter@hashicorp.com>

services: cr followup

ed37d21

services: fix imports

2443174

shoenig force-pushed the f-arbitrary-addresses branch from 470ed8a to 2443174 Compare April 22, 2022 14:16

vercel bot deployed to Preview – nomad-storybook-and-ui April 22, 2022 14:18 View deployment

shoenig merged commit c8bd090 into main Apr 22, 2022

shoenig deleted the f-arbitrary-addresses branch April 22, 2022 14:34

shoenig mentioned this pull request May 11, 2022

Allow setting tagged addresses to services #12950

Closed

shoenig mentioned this pull request Aug 12, 2022

Allow manually specifying IP address for service #8140

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

services: enable setting arbitrary address value in service registrations #12720

services: enable setting arbitrary address value in service registrations #12720

shoenig commented Apr 20, 2022 •

edited

Loading

schmichael left a comment

schmichael Apr 21, 2022

schmichael Apr 21, 2022

schmichael Apr 21, 2022

schmichael Apr 21, 2022

jrasell Apr 22, 2022

schmichael Apr 21, 2022

schmichael Apr 21, 2022

jrasell Apr 22, 2022

shoenig Apr 22, 2022

jrasell left a comment

github-actions bot commented Oct 15, 2022

		// service or check registration. If no port label is specified (an empty value),
		// zero values are returned because no address could be resolved.

services: enable setting arbitrary address value in service registrations #12720

services: enable setting arbitrary address value in service registrations #12720

Conversation

shoenig commented Apr 20, 2022 • edited Loading

Basic Example

Interpolation Example

Example with Consul Checks

Bonus EC2 Public IPv4 Example

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrasell left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 15, 2022

shoenig commented Apr 20, 2022 •

edited

Loading