CLI Allow querying all namespaces for jobs and allocations

[notnoop]

This allows an Enterprise operator to query for lists in jobs and allocations across all namespaces they have capabilities in.

UX

When -all-namespaces command line argument is set, or if NOMAD_ALL_NAMESPACES env var is set to truthy value (e.g. 1, true), nomad job and nomad alloc commands will lookup relevant jobs/allocations in all namespaces they are authorized to.

nomad job status -all-namespaces will emit all the jobs across namespaces (with a namespace column added). nomad job status -all-namespaces foo will return the job info for foo job regardless of which namespace is in; if there multiple foo jobs, the CLI will emit back an error message, indicating the possible values.

# namespace specific status
$ nomad job status
ID        Type   Priority  Status   Submit Date
example1  batch  50        running  2020-06-17T10:49:26-04:00

# list jobs across all namespaces
$ nomad job status -all-namespaces
ID        Namespace  Type   Priority  Status   Submit Date
example1  default    batch  50        running  2020-06-17T10:49:26-04:00
example1  qa         batch  50        running  2020-06-17T10:49:37-04:00
qa-job    qa         batch  50        running  2020-06-17T10:50:28-04:00

# get job status for a job in a specific namespace
$ nomad job status -short ex
ID            = example1
Name          = example1
Submit Date   = 2020-06-17T10:49:26-04:00
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

# when checking multiple namespaces, report an error message if multiples are found
$ nomad job status -short -all-namespaces ex
Prefix matched multiple jobs

ID        Namespace  Type   Priority  Status   Submit Date
example1  default    batch  50        running  2020-06-17T10:49:26-04:00
example1  qa         batch  50        running  2020-06-17T10:49:37-04:00

# but namespace can always be specified to disambiguate
$ nomad job status -short -namespace qa ex
ID            = example1
Name          = example1
Submit Date   = 2020-06-17T10:49:37-04:00
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = qa
Status        = running
Periodic      = false
Parameterized = false

# querying with a prefix if a uniue job is found works
$ nomad job status -short -all-namespaces qa
ID            = qa-job
Name          = qa-job
Submit Date   = 2020-06-17T10:50:28-04:00
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = qa
Status        = running
Periodic      = false
Parameterized = false

Abandoned ideas

Defaulting to -all-namespaces

I considered avoiding the additional argument and defaulting to all namespaces if no --namespace is specified. Such change would be a significant change in behavior that will surprise users.

Using a sentinel namespace value

We could reserve a sentinel namespace value (e.g. --namespace global) to indicate querying all namespaces. However, operators currently can create any namespace, and introducing a special reserved value might collide with an existing namespace and result into confusion.

Implementation Details

The implementation has three main components:

The RPC Handlers for Job.List and Alloc.List accept AllNamespaces argument to indicate whether one should search all namespaces the request token has access to. This PR implements Allocations.List, as Job.List was already updated in #8001 .

Then the API and CLI components need to pass AllNamespaces appropriately. Due to the sheer number of endpoints this affects, I made the flags a global Client Meta/Config options.

The CLI subcommands typically adhere to the following implementation pattern:

1. attempt to find all job/allocation matches with given prefix - passing AllNamespaces as appropriate
2. If a job is found, do a specific lookup while passing the namespace found earlier.

As such, {Job|Alloc}.Get RPC endpoints aren't modified, and as before must require the correct namespace being passed.

Incidentally, we didn't need to add any indexes to the state store to accomodate this. Alloc.List already isn't indexed by namespaces, so checking multiple namespaces is roughly the same computationally intensive operation as the namespaced ones.

Potential follow ups

• We could implement the same logic for Evals as well but I'm uncertain if it's needed, so waiting for some customer feedback.
• Add tests in the private Enterprise repository. Hard to test this in OSS where namespaces aren't supported.

[drewbailey]

Any reason not to add this into s.parse?

[drewbailey]

Ah, looking closer into parse it doesn't make as much sense, nvm

oops, wrong pr

[notnoop]

Superseded by #8192

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.