consul: correctly check consul acl token namespace when using consul oss #10720

shoenig · 2021-06-07T20:03:54Z

This PR fixes the Nomad Object Namespace <-> Consul ACL Token relationship
check when using Consul OSS (or Consul ENT without namespace support).

Nomad v1.1.0 introduced a regression where Nomad would fail the validation
when submitting Connect jobs and allow_unauthenticated set to true, with
Consul OSS - because it would do the namespace check against the Consul ACL
token assuming the "default" namespace, which does not work because Consul OSS
does not have namespaces.

Instead of making the bad assumption, expand the namespace check to handle
each special case explicitly.

Fixes #10718

notnoop

Few questions; approach seems reasonable.

notnoop · 2021-06-08T13:54:48Z

nomad/consul_policy.go

+	case namespace == token.Namespace:
+		// ACLs enabled, namespaces are the same
+		return nil


I find it a bit easier if the common cases are grouped together, and special cases are together. I'd suggest moving this case to be the second.

notnoop · 2021-06-08T14:03:36Z

nomad/consul_policy.go

+		// provide a more informative error message.
+		return errors.Errorf("consul ACL token requires using namespace %q", token.Namespace)
+
+	default:


I found it a tad harder to reason through conditions, maybe because of ordering. I would probably find it easier if the conditions were order the following way:

namespace == token.Namespace: either ACL not enabled, or it's enabled and they match

token.Namespace == "default": special case

token.Namespace != "default" && namespace == "" <- token.Namespace != "default" is redundant here but included for explicitness

default: the mismatch case

Thanks for this suggestion, this is easier to read 🙂

notnoop · 2021-06-08T14:06:42Z

nomad/consul_policy.go

+		// ACLs enabled, must defer to per-object checking, since the token could
+		// have namespace_prefix blocks allowing the operation


I'm not familiar with namespace_prefix. Is it only applicable if token's namespace is default? Why doesn't it apply in the default or token.Namespace != "default" cases? Is a special case for handling legacy apps only? Might be useful to add more context.

Yep, Consul token ACL policies can supply namespace or namespace_prefix blocks which give the token permission beyond the "default" namespace. Those blocks are only allowable in tokens in the "default" namespace.

https://www.consul.io/docs/security/acl/acl-rules#namespace-rules

Expanded on the comment to clarify.

This PR fixes the Nomad Object Namespace <-> Consul ACL Token relationship check when using Consul OSS (or Consul ENT without namespace support). Nomad v1.1.0 introduced a regression where Nomad would fail the validation when submitting Connect jobs and allow_unauthenticated set to true, with Consul OSS - because it would do the namespace check against the Consul ACL token assuming the "default" namespace, which does not work because Consul OSS does not have namespaces. Instead of making the bad assumption, expand the namespace check to handle each special case explicitly. Fixes #10718

shoenig · 2021-06-08T19:25:12Z

@notnoop I applied your recommendations, thanks for the suggestion.

I also fixed the failing tests, which then made me realize we need to split the tests between oss and ent - they were originally written around the same bad assumption that caused this bug in the first place. So now we also have new files

nomad/consul_oss_test.go
nomad/consul_policy_oss_test.go
nomad/job_endpoint_oss_test.go

which are oriented around the fact that Nomad OSS does not consider the group.consul.namespace field. I'm working on a followup PR for ENT which expands on the same tests but with all combinations of possible input for group.consul.namespace and the Consul token Namespace.

notnoop

Looks great. Awesome tests. LGTM.

notnoop · 2021-06-08T20:02:46Z

nomad/consul_oss_test.go

+	t.Run("check-permissions kv read", func(t *testing.T) {
+		t.Run("uses kv has permission", func(t *testing.T) {


I don't know how I personally feel about this style of nested t.Run: When the assertions are simply delegation to try, it feels a simple table tests might read better and is more concise.

Yeah, for tests like these it's hard to find the right balance between readability of the code and readability of the description of the test. By composing t.Run we get a free way of composing the description, making it obvious what the difference is between one particular test from the tests surrounding it. OTOH it's a lot of lines of code that aren't actually doing anything. I'll take a look at refactoring these tests some more after the bugfix release.

shoenig · 2021-06-09T14:56:35Z

Spot checking (oss Nomad + oss Consul)

$ nomad job run api.nomad
Error submitting job: Unexpected response code: 500 (job-submitter consul token denied: missing consul token)
$ nomad job run -consul-token $(uuid-tool) api.nomad
Error submitting job: Unexpected response code: 500 (job-submitter consul token denied: unable to read consul token: Unexpected response code: 403 (ACL not found))
$ nomad job run -consul-token 76221243-1631-e241-9b34-ffbcb6c36025 api.nomad
==> Monitoring evaluation "3d2dcf7f"
    Evaluation triggered by job "api"
    Evaluation within deployment: "8afe1793"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "3d2dcf7f" finished with status "complete"

github-actions · 2022-11-20T02:32:05Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

notnoop added this to the 1.1.1 milestone Jun 8, 2021

notnoop reviewed Jun 8, 2021

View reviewed changes

shoenig force-pushed the f-cns-acl-check branch from bdf2227 to af9d1d4 Compare June 8, 2021 16:43

vercel bot deployed to Preview – nomad-storybook-and-ui June 8, 2021 16:43 View deployment

vercel bot temporarily deployed to Preview – nomad June 8, 2021 16:43 Inactive

shoenig force-pushed the f-cns-acl-check branch from af9d1d4 to a5f908f Compare June 8, 2021 17:38

vercel bot temporarily deployed to Preview – nomad June 8, 2021 17:38 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui June 8, 2021 17:38 View deployment

shoenig force-pushed the f-cns-acl-check branch from a5f908f to 09c9a17 Compare June 8, 2021 18:56

vercel bot deployed to Preview – nomad-storybook-and-ui June 8, 2021 18:56 View deployment

vercel bot temporarily deployed to Preview – nomad June 8, 2021 18:56 Inactive

shoenig marked this pull request as ready for review June 8, 2021 19:25

shoenig requested review from notnoop and tgross June 8, 2021 19:25

notnoop approved these changes Jun 8, 2021

View reviewed changes

shoenig merged commit 402b19c into main Jun 8, 2021

shoenig deleted the f-cns-acl-check branch June 8, 2021 20:43

github-actions bot locked as resolved and limited conversation to collaborators Nov 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

consul: correctly check consul acl token namespace when using consul oss #10720

consul: correctly check consul acl token namespace when using consul oss #10720

shoenig commented Jun 7, 2021

notnoop left a comment

notnoop Jun 8, 2021

notnoop Jun 8, 2021

shoenig Jun 8, 2021

notnoop Jun 8, 2021

shoenig Jun 8, 2021 •

edited

Loading

shoenig commented Jun 8, 2021

notnoop left a comment

notnoop Jun 8, 2021

shoenig Jun 8, 2021

shoenig commented Jun 9, 2021

github-actions bot commented Nov 20, 2022

		// ACLs enabled, must defer to per-object checking, since the token could
		// have namespace_prefix blocks allowing the operation

		t.Run("check-permissions kv read", func(t *testing.T) {
		t.Run("uses kv has permission", func(t *testing.T) {

consul: correctly check consul acl token namespace when using consul oss #10720

consul: correctly check consul acl token namespace when using consul oss #10720

Conversation

shoenig commented Jun 7, 2021

notnoop left a comment

Choose a reason for hiding this comment

notnoop Jun 8, 2021

Choose a reason for hiding this comment

notnoop Jun 8, 2021

Choose a reason for hiding this comment

shoenig Jun 8, 2021

Choose a reason for hiding this comment

notnoop Jun 8, 2021

Choose a reason for hiding this comment

shoenig Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

shoenig commented Jun 8, 2021

notnoop left a comment

Choose a reason for hiding this comment

notnoop Jun 8, 2021

Choose a reason for hiding this comment

shoenig Jun 8, 2021

Choose a reason for hiding this comment

shoenig commented Jun 9, 2021

github-actions bot commented Nov 20, 2022

shoenig Jun 8, 2021 •

edited

Loading