consul: allow stale namespace results

[schmichael]

Nomad reconciles services it expects to be registered in Consul with
what is actually registered in the local Consul agent. This is necessary
to prevent leaking service registrations if Nomad crashes at certain
points (or if there are bugs).

When Consul has namespaces enabled, we must iterate over each available
namespace to be sure no services were leaked into non-default
namespaces.

Since this reconciliation happens often, there's no need to require
results from the Consul leader server. In large clusters this creates
far more load than the "freshness" of the response is worth.

Therefore this patch switches the request to AllowStale=true

[shoenig]

@schmichael is this ready to review or should still be in draft? If it's ready, can you rebase with main to fix the GHA runners?

[shoenig]

This change seems like a great improvement. We could probably also take it up a notch by caching the namespace list results in NamespaceClient.List, like we already for for caching the SKU/enabled result.

[schmichael]

@shoenig recorded that #13035 for future work.

[shoenig]

LGTM!

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.