(PCUI = 2024.10.0) PCUI Lambda stops working because ECR policy gets deleted on PCUI stack update

The issue

We have discovered an issue that causes the PCUI Lambda function to stop working when the PCUI stack is updated. In particular, when the PCUI stack is updated, the policy associated with the ECR repository is unexpectedly removed, making it impossible for the PCUI Lambda function to load the code from it. This happens regardless of the content of the update. If your PCUI stack is affected by this issue you will get an internal server error when trying to access PCUI and the below error is returned by the Lambda Console:

Failed to restore the function ParallelClusterUIFun-xxxxxxxxxx The function does not have permission to access the specified image

Affected versions (OSes, schedulers)

• All PCUI versions <= 2024.10.0

Mitigation

To mitigate this issue you can do one of the following:

1. Redeploy the Lambda function after every PCUI stack update to make AWS Lambda reset the ECR policy. This can be done though AWS Lambda console or with AWS CLI command update-function-code.
2. Manually add the required policy to the ECR repo.

RepositoryPolicyText:
  Version: 2012-10-17
  Statementon: 2012-10-17
  Statement:
    - Sid: AllowLambdaToRetrieveECRImages
      Effect: Allow
      Principal:
        Service: lambda.amazonaws.com
      Action:
        - ecr:BatchGetImage
        - ecr:DeleteRepositoryPolicy
        - ecr:GetDownloadUrlForLayer
        - ecr:GetRepositoryPolicy
        - ecr:SetRepositoryPolicy
      Condition:
        StringLike:
          aws:Sou:
    - Sid: AllowLambdaToRetrieveECRImages
      Effect: Allow
      Principal:
        Service: lambda.amazonaws.com
      Action:
        - ecr:BatchGetImage
        - ecr:DeleteRepositoryPolicy
        - ecr:GetDownloadUrlForLayer
        - ecr:GetRepositoryPolicy
        - ecr:SetRepositoryPolicy
      Condition:
        StringLike:
          aws:SourceArn: !Sub arn:<PARTITION>:lambda:<REGION>:<ACCOUNT_ID>:function:*