(3.0.0 3.1.4) Unable to perform cluster update when using API or documented user policies

The issue

Cluster updates after changing the LambdaFunctionsRole using the ParallelCluster API or CLI fails with the cluster stack in UPDATE_FAILED status and reason

User: arn:aws:sts::xxx:assumed-role/yyy is not authorized to perform: lambda:ListTags on resource: zzz

This happens when updating the property LambdaFunctionsRole during a cluster update using the ParallelCluster API from version 3.0.0 to 3.1.5 or any CLI version when the user has an IAM role with the user policies documented in the official doc.

Affected versions

• ParallelCluster API from 3.0.0 to 3.1.5 included
• Any version of ParallelCluster CLI, when using IAM role with attached user policies documented in the official doc (version published before May 20, 2022)

Mitigation

When using any version of the ParallelCluster CLI, if you are using an IAM role with user policies described in the official doc, be sure that it contains the lambda:ListTags, lambda:TagResource, and lambda:UntagResource policies.

When using the ParallelCluster API from version 3.0.0 to 3.1.5, please follow one of the mitigation paths listed below depending on if you have already an API stack deployed or you want to create a new API stack from scratch.

Update Existing API Stack for version 3.1.4/3.1.5

Update an existing API stack with the following command, where the --template-url is different from the one documented in the official doc.

REGION=<region>
API_STACK_NAME=<stack-name>  # This needs to correspond to the existing API stack name
aws cloudformation update-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \  
    --template-url https://us-east-1-aws-parallelcluster.s3.us-east-1.amazonaws.com/patches/3.1.4/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
aws cloudformation wait stack-update-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Create new API Stack for version 3.1.4/3.1.5

If you need to deploy a new API stack use the following command which supersedes the one documented in the official doc.

REGION=<region>
API_STACK_NAME=<stack-name>  # This can be any name
aws cloudformation create-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \
    --template-url https://us-east-1-aws-parallelcluster.s3.us-east-1.amazonaws.com/patches/3.1.4/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
aws cloudformation wait stack-create-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Update Existing API Stack for versions from 3.0.0 to 3.1.3

Update the existing IAM user role deployed through the API stack with the following manual steps to be performed through the AWS console:

• Identify the IAM role resource with Logical ID named ParallelClusterUserRole deployed by the API stack
• Add the actions lambda:ListTags, lambda:TagResource and lambda:UntagResource to the policy whose name starts with pcluster-api-ParallelClusterClusterPolicy- and save the changes