support multiple host_network names for the same interface

[AndrewChubatiuk]

Nomad version

nomad v1.0.4

Issue

Having issue with multi-region infrastructure setup. One region is in AWS VPC and another is in Hetzner + Nebula for private networks setup. Host networks configuration in Hetzner is:

{
  "host_network": {
    "private": {
      "cidr": "<nebula-ip-address>/32"
    },
    "public": {
      "cidr": "<hetzner-ip-address>/32"
    }
  }
}

To be able to use the same job for both regions (cause there's no host_network variable interpolation) had to use the same interface for both public and private agent networks configuration in AWS for compatibility

{
  "host_network": {
    "private": {
      "cidr": "<awsvpc-ip-address>/32"
    },
    "public": {
      "cidr": "<awsvpc-ip-address>/32"
    }
  }
}

But nomad doesn't create allocations in AWS a job, which uses both public and private interfaces, cause nomad agent is registred with private network only. Is there any option to either add host_network interpolation support or use multiple network names for the same interface?

[tgross]

Hi @AndrewChubatiuk!

But nomad doesn't create allocations in AWS a job, which uses both public and private interfaces, cause nomad agent is registred with private network only.

I'm not sure I understand... can you explain the behavior you're seeing here a bit more?

[AndrewChubatiuk]

@tgross
I have a problem when trying to apply a system job in a multiregion setup were regions have different host network configurations.
In AWS region there is a private AWS VPC network interface and in a region that is in Hetzner there are public (default instance IP address) and private (Nebula) network interfaces.
To apply a system job which should use a private network in AWS and public in Hetzner I've created 2 PRs with possible solutions:

1. host_network value interpolation in a port stanza. In this scenario host network value can be configured in Nomad Agent meta. Nomad Agent configuration and job example:

Hetzner region Nomad Agent config example

{
   "client":{
      "host_network":{
         "public":{
            "cidr":"<public-ip>"
         },
         "private":{
            "cidr":"<nebula-ip>"
         }
      },
      "meta":{
         "public_network":"public"
         ...
      }
      ...
   }
}

AWS region Nomad Agent config example

{
   "client":{
      "host_network":{
         "private":{
            "cidr":"<aws-vpc-ip>"
         }
      },
      "meta":{
         "public_network":"private"
         ...
      }
      ...
   }
}

Job Example

job "example" {
  datacenters = ["hetzner", "aws"]
  type        = "system"
  group "example" {
    network {
      mode = "bridge"
      port "https" {
        static = 9090
        to       = 9090
        host_network = "${meta.public_network}"
      }
    }
    ...
  }
}

2. Add ability to set multiple aliases for a host network. Configuration example:

Hetzner region Nomad Agent config example

{
   "client":{
      "host_network":{
         "public":{
            "cidr":"<public-ip>"
         },
         "private":{
            "cidr":"<nebula-ip>"
         }
      }
      ...
   }
}

AWS region Nomad Agent config example

{
   "client":{
      "host_network":{
         "private":{
            "cidr":"<aws-vpc-ip>"
         },
         "public":{
            "cidr":"<aws-vpc-ip>"
         }
      }
      ...
   }
}

Job Example

job "example" {
  datacenters = ["hetzner", "aws"]
  type        = "system"
  group "example" {
    network {
      mode = "bridge"
      port "https" {
        static = 9090
        to       = 9090
        host_network = "public"
      }
    }
    ...
  }
}

[groggemans]

I'm also running into this issue, but in a slightly different way.

On several of my test nodes I have a large amount of host networks. To properly identify them they are all configured as host_network in the agent config. As a result I also set a fixed name for what otherwise would be the "default" network range.

However that default range is different between some of the nodes. As a result I have no way to schedule a job in the "default" network, as that is no longer known to Nomad. Using the host_network name instead would result in the problem outlined by Andrew above.

For my use case there would be another solution besides those mentioned by Andrew; And that is to let Nomad also define the "default" host network, even if a host_network with the same range is defined. This would also work in Andrews case, but I do agree that defining the same range with multiple names or interpolating the name would be cleaner for that use case.

Losing the "default" range when making all host networks explicit also results in jobs being unable to schedule if no explicit host_network is defined for a port (also for connect sidecars). An inconvenience for me, but could be used to enforce more explicit network definitions for others. Multiple aliases or interpolation is probable the better solution as it still suits both behaviors.

[tgross]

Closed by #10104, which will ship in Nomad 1.1.0

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.