(3.7.0‐3.8.0) ParallelCluster API Deployment fails due to IAM Policy size exceeding service limits

Bug description

The deployment of the ParallelCluster API stack fails when the parameter Region is set. In particular, CloudFormation stack fails with CREATE_FAILED and the message:

Resource handler returned message: "Cannot exceed quota for PolicySize: 6144 (Service: Iam, Status Code: 409, Request ID: bb3fa361-b2a7-41d9-be2b-d7668ee8ba1f)" (RequestToken: 7b853345-c08e-555d-ee3a-f163521a0acc, HandlerErrorCode: ServiceLimitExceeded)

The issue is caused by the Region parameter that is used in the IAM managed policy ParallelClusterClusterPolicy. This addition makes the policy exceed the maximum length allowed.

Affected versions (OSes, schedulers)

• ParallelCluster API 3.7.0
• ParallelCluster API 3.8.0

Mitigation

To successfully deploy ParallelCluster API, set/keep the parameter Region to its default value * , which allows to deploy clusters in every region.

If the API scope needs to be restricted to a specific region, the following procedure can be followed to create a Permissions boundary and attach it to the ParallelClusterLambdaRole IAM role. You can find more information on this IAM role in the official ParallelCluster documentation here.

1. Create the permissions boundary policy below replacing <YOUR_REGION> with the region where you want to restrict access.

   1. In a shell access your AWS account in the region where you have deployed the API stack

   2. Save in a file called PCAPI-<YOUR_REGION>-boundary.json with the content below

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": "*",
                  "Resource": "*",
                  "Condition": {
                      "StringEquals": {
                          "aws:RequestedRegion": "<YOUR_REGION>"
                      }
                  }
              }
          ]
      }
      

      Add the policy with the following command

      aws iam create-policy --policy-name PCAPI-<YOUR_REGION>-boundary --policy-document file://PCAPI-<YOUR_REGION>-boundary.json
      

      Expected output like:

      {
          "Policy": {
              "PolicyName": "PCAPI-eu-west-1-boundary",
              "PolicyId": "ANPAUYRLW327RZ6KGKAQG",
              "Arn": "arn:aws:iam::123456789012:policy/PCAPI-eu-west-1-boundary",
              "Path": "/",
              "DefaultVersionId": "v1",
              "AttachmentCount": 0,
              "PermissionsBoundaryUsageCount": 0,
              "IsAttachable": true,
              "CreateDate": "2024-02-29T13:30:50+00:00",
              "UpdateDate": "2024-02-29T13:30:50+00:00"
          }
      }
      

      Resource: AWS guide to create the permission boundary - [Creating a Permissions Boundary](https://docs.aws.amazon.com/prescriptive-guidance/latest/transitioning-to-multiple-aws-accounts/ creating-a-permissions-boundary.html)

2. Identified the role ParallelClusterLambdaRole with the following procedure:
   1. In a shell access your AWS account in the region where you have deployed the API stack
   2. Execute the following commands replacing <YOUR_API_STACK_NAME> with the stack name of your API.

      POLICY_STACK=$(aws cloudformation describe-stack-resource --logical-resource-id PclusterPolicies --stack-name <YOUR_API_STACK_NAME> --query 'StackResourceDetail.PhysicalResourceId' --output text | awk -F / '{print $2}')
      aws cloudformation describe-stack-resources --logical-resource-id ParallelClusterLambdaRole --stack-name $POLICY_STACK --query 'StackResources[*].PhysicalResourceId' --output text
      

      Expected output like:

      ParallelClusterLambdaRole-c503c760
      

3. Modify the role attaching the permissions boundary.
   1. In a shell access your AWS account in the region where you have deployed the API stack
   2. Execute the following commands replacing <YOUR_PCAPI_ROLE> with the role name returned from the previous command and <YOUR_REGION> with the region where you want to restrict access.

      PERM_BOUNDARY_ARN=$(aws iam list-policies --query 'Policies[?PolicyName==`PCAPI-<YOUR_REGION>-boundary`].Arn' --output text)
      aws iam put-role-permissions-boundary --role-name <YOUR_PCAPI_ROLE> --permissions-boundary $PERM_BOUNDARY_ARN
      aws iam get-role --role-name <YOUR_PCAPI_ROLE> --query "Role.PermissionsBoundary.PermissionsBoundaryArn" --output text
      

      Expected output like:

      arn:aws:iam::123456789012:policy/PCAPI-eu-west-1-boundary
      

      Resource: AWS guide to attach the permission boundary - Attaching a Permissions Boundary