Controller changes to improve UX for setting up TGWs on ECS

[Ganeshrockz]

Changes proposed in this PR:

Background

In an ACL enabled Consul cluster where terminating gateway based ECS workloads are deployed, the ACL token used by the TGW proxy needs service:write permission on all the referenced services in the gateway config. This token will be used to pull in relevant information about those services from Consul's catalog. For example, let's consider a case where an example-client-app service (present in the mesh) tries to reach to an external example-server-app service present outside the mesh through a TGW. The terminating gateway proxy in this case needs a token with the following policy

service "example-server-app" {
   policy = "write"
}

for the E2E to work. Without making any changes to the ECS controller, the current UX looks like this

1. Users register their external services to Consul's catalog.
2. Users add the name of their external service to the terminating gateway's config entry.
3. Users would then need to add the ☝️ policy to the token that gets created for the terminating gateway's service via the AWS IAM auth method.

The first 2 steps can be automated with terraform but the 3rd one has to be done manually by the operator because it is difficult to get the accessor ID of the token. Only with an accessorID in place will the operator be able to attach a policy to the token.

One possible workaround is to create a wildcard policy that looks something like

service "" {
  policy = "write"
}

and add a binding rule to the configured auth method via the ECS controller that makes sure to add this policy to the resulting token for the TGW service identity. While this should technically work, it doesn't follow the principle of least privileges.

Proposed solution

1. On startup, make the controller create a dummy role without any policies named consul-ecs-terminating-gateway-role
2. Add a binding rule to the auth method, that attaches the previously created role in step 1 to the resulting token.
3. Users looking to register external services could automate the policy creation and perform the policy attachment to the role via terraform. A sample terraform looks something like

# Create the policy
resource "consul_acl_policy" "external_server_app_policy" {
  name  = "external_server_app_write_policy"
  rules = <<-RULE
    service "${var.name}-external-server-app" {
      policy = "write"
    }
    RULE

  provider = consul.dc1-cluster
}

# Read the pre-created role by the controller
data "consul_acl_role" "ecs_terminating_gateway_default_role" {
  depends_on = [module.ecs_controller]
  name       = "consul-ecs-terminating-gateway-role"

  provider = consul.dc1-cluster
}

# Attach the policy to the role
resource "consul_acl_role_policy_attachment" "external_server_app_role_policy_attachment" {
  role_id = data.consul_acl_role.ecs_terminating_gateway_default_role.id
  policy  = consul_acl_policy.external_server_app_policy.name

  provider = consul.dc1-cluster
}

This way the token gets the required permissions to pull in details about the referenced external services. This also reduces manual intervention by operators.

How I've tested this PR:

How I expect reviewers to test this PR:

Checklist:

• Tests added
• CHANGELOG entry added