Scan for and reserve already-used ports when nomad starts

[cbednarski]

Ports are dynamically allocated in a range between 20000 and 60000. We should scan and detect ports in this range that are already bound so we don't collide with other services. The high port range is likely safe, but we should check.

Also, we should check for low ports in case a service tries to use a reserved port in that range.

[cetex]

What if something launches outside of nomad after nomad has "scanned" the range of ports?

I'd say the best solution is to define ephemeral ports in sysctl and then setup nomad to use ports outside that defined range when assigning ports to services.

If you're using ports from the ephemeral range when assigning ports to services you're gonna have collisions regularly on most servers in a high-traffic environment.

[legal90]

Wasn't it implemented in #494, by the way ?

[dadgar]

@legal90: Not quite. The idea was when the Nomad Client boots it would scan and figure out what ports were already being used and then reserve them

[c4milo]

@dadgar not that this is a problem I would have as the nodes I provisioned are exclusively dedicated to Nomad. But, what's the plan for keeping track of ports getting used outside of Nomad once Nomad Client is already up?

[diptanu]

@c4milo Usually the way this works is that operators tell the range of ports the cluster scheduler is allocated on every interface on the node. So everything else which runs on the Node, the operator has to make sure they don't run on the ports allocated to the scheduler.

And if a process is not allowed to bind on a port it's not allocated, I have seen certain schedulers kill them as that's the only way to do fare share for port resources or the task is put into a network namespace.

[c4milo]

@diptanu fair enough.

[tugbabodrumlu]

Do you plan to have this capability soon?

[dadgar]

@tugbabodrumlu It isn't super high priority for us since you can reserve the ports in the client config using this: https://www.nomadproject.io/docs/agent/configuration/client.html#reserved

And most ports on the host should be managed by Nomad. Hope that helps!

[schmichael]

I'm going to consider #11167 the solution here and close this vintage issue.

tl;dr - I'm not convinced the costs this feature would incur are worth it for the benefits, so closing.

Fingerprinting used ports is tricky because they may not be used for long! So do we repeatedly fingerprint and constantly update node objects to servers? This could cause a lot of extra load on Nomad servers for clients that accidentally (or intentionally!) overlap Nomad's dynamic port range and their OS's ephemeral port range! Even for long-lived services periodic fingerprinting is very problematic: what if a long-lived service is taken down for maintenance? If Nomad sees the port as free and schedules a service on it, you now prevent the original service from starting back up when maintenance is complete! To make matters worse you have an arbitrary service listening on the port and potentially serving traffic to clients intending to reach the down service!

Fingerprinting once on startup is also tricky because it may appear to work as expected (eg the first time it's enabled it may properly exclude a port in use by another long-lived service), but then on system restart Nomad may start first and miss that inuse port and mark it available. Operators would have to carefully configure all services to start in the right order and block until ports are actually bound (instead of just process startup). It seems far easier to simply manually configure port reservations/ranges in Nomad.

While it's true none of the issues outlined above are intractable, I have a hard time believing solving them is worth it. As of 1.2 Nomad has 2 easy to understand and tune configuration parameters to address this use case. While manual configuration is unfortunate, it avoids all of the issues above with a fairly minimal level of effort.

Hopefully most, if not all, other daemons running on Nomad nodes can be managed by Nomad directly making this feature entirely moot! I'd rather spend engineering effort solving for more use cases than trying to figure out how to make this fingerprinting foolproof.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.