Improve dynamic port selection

[bagelswitch]

The way that dynamic port assignment is done currently doesn't work well at all if significant ranges of reserved_ports are specified in nomad client config. For example:

client {
reserved {
reserved_ports = "23000-65535"
}
}

With this config (which leaves ports 20000-22999 available for assignment), running a job requiring several dynamic ports will fail more often than not, as a random pick in the range 20000-60000 will fall in a reserved range >90% of the time.

This change ensures that acceptable ports are always assigned, so long as a sufficient number of free ports exist in any non-reserved range between MinDynamicPort (20000) and MaxDynamicPort (60000).

[dadgar]

I think it may be simpler if we do the computation as follows:

a := []Ports{Port Range - node.Reserved - ask.Reserved}
shuffle(a)
dynamic ports := a[:len(ask.DynamicPorts)]

I think this code will be simpler and have a better run time cost. And then we can remove the maxAttempts

[bagelswitch]

Thanks @dadgar - something more like this?

[dadgar]

Closing in favor of #1526

Thanks for finding and bringing this up!

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.