Missing permissions prevent EFS CSI from creating PersistentVolumeClaim/PersistentVolume

[jamiemo]

Description

As per the example EKS Cluster for Stateful Workloads when creating a PVC it fails to mount or create the PV/EFS Access Point and is stuck in the Pending state.

This is due to missing police as outlined in Create an IAM policy and role and the associated policy document. Specifically:

    {
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:TagResource"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
        }
      }
    }

Workaround

This workaround provides the additional missing policy.

module "eks_blueprints_kubernetes_addons" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.24.0"
  ...
  enable_aws_efs_csi_driver            = true
  aws_efs_csi_driver_irsa_policies     = [resource.aws_iam_policy.aws_efs_csi_driver_tags.arn]
  ...
}

resource "aws_iam_policy" "aws_efs_csi_driver_tags" {
  name        = "${module.eks_blueprints.eks_cluster_id}-efs-csi-tag-policy"
  description = "IAM Policy for AWS EFS CSI Driver Tags"
  policy      = data.aws_iam_policy_document.aws_efs_csi_driver_tags.json
  tags        = local.tags
}

data "aws_iam_policy_document" "aws_efs_csi_driver_tags" {
  statement {
    sid       = "AllowTagResource"
    effect    = "Allow"
    resources = [
      module.efs.arn
    ]
    actions   = ["elasticfilesystem:TagResource"]

    condition {
      test     = "StringLike"
      variable = "aws:ResourceTag/efs.csi.aws.com/cluster"
      values   = ["true"]
    }
  }
}

If your request is for a new feature, please use the Feature request template.

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: v4.24.0

• Terraform version: v1.4.6

• Provider version(s):

• provider registry.terraform.io/gavinbunney/kubectl v1.14.0
• provider registry.terraform.io/hashicorp/aws v4.63.0
• provider registry.terraform.io/hashicorp/cloudinit v2.3.2
• provider registry.terraform.io/hashicorp/helm v2.9.0
• provider registry.terraform.io/hashicorp/kubernetes v2.19.0
• provider registry.terraform.io/hashicorp/local v2.4.0
• provider registry.terraform.io/hashicorp/null v3.2.1
• provider registry.terraform.io/hashicorp/random v3.5.1
• provider registry.terraform.io/hashicorp/time v0.9.1
• provider registry.terraform.io/hashicorp/tls v4.0.4
• provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Reproduction Code [Required]

# EFS storage class for persistent volumes
resource "kubernetes_storage_class_v1" "efs" {
  metadata {
    name = "efs"
  }

  storage_provisioner = "efs.csi.aws.com"
  parameters = {
    provisioningMode = "efs-ap" # Dynamic provisioning
    fileSystemId     = module.efs.id
    directoryPerms   = "700"
  }

  mount_options = [
    "iam"
  ]

  depends_on = [
    module.eks_blueprints_kubernetes_addons
  ]
}

module "efs" {
  source  = "terraform-aws-modules/efs/aws"
  version = "~> 1.0"

  creation_token = local.name
  name           = local.name

  # Mount targets / security group
  mount_targets = {
    for k, v in zipmap(local.azs, module.vpc.private_subnets) : k => { subnet_id = v }
  }
  security_group_description = "${local.name} EFS security group"
  security_group_vpc_id      = module.vpc.vpc_id
  security_group_rules = {
    vpc = {
      # relying on the defaults provdied for EFS/NFS (2049/TCP + ingress)
      description = "NFS ingress from VPC private subnets"
      cidr_blocks = module.vpc.private_subnets_cidr_blocks
    }
  }

  tags = local.tags
}

Steps to reproduce the behavior:

Expected behaviour

$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
efs-claim-1   Bound    pvc-d102361a-fb18-44b5-a05b-9eabfe933b58   5Gi        RWX            efs            7s

$ kubectl describe pvc efs-claim-1
Name:          efs-claim-1
Namespace:     default
StorageClass:  efs
Status:        Bound
Volume:        pvc-d102361a-fb18-44b5-a05b-9eabfe933b58
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
               volume.kubernetes.io/storage-provisioner: efs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       efs-app-1
Events:
  Type    Reason                 Age   From                                                                                      Message
  ----    ------                 ----  ----                                                                                      -------
  Normal  ExternalProvisioning   52s   persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator
  Normal  Provisioning           52s   efs.csi.aws.com_efs-csi-controller-575b89d87d-vv8g7_cc810a13-232f-43e0-a9c1-e8de0a178b72  External provisioner is provisioning volume for claim "default/efs-claim-1"
  Normal  ProvisioningSucceeded  52s   efs.csi.aws.com_efs-csi-controller-575b89d87d-vv8g7_cc810a13-232f-43e0-a9c1-e8de0a178b72  Successfully provisioned volume pvc-d102361a-fb18-44b5-a05b-9eabfe933b58

$ kubectl get pv                  
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                 STORAGECLASS   REASON   AGE
pvc-0a2b650d-9f2e-42b8-8f29-9b26783bbaa7   32Gi       RWO            Delete           Bound    kubecost/kubecost-cost-analyzer       gp3                     22m
pvc-9bd82ee9-b913-43a8-88c6-ae09ab9ba77d   32Gi       RWO            Delete           Bound    kubecost/kubecost-prometheus-server   gp3                     22m
pvc-d102361a-fb18-44b5-a05b-9eabfe933b58   5Gi        RWX            Delete           Bound    default/efs-claim-1                   efs                     91s

Actual behaviour

$ kubectl get pvc
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
efs-claim-1   Pending                                      efs            13s

$ kubectl describe pvc efs-claim-1
Name:          efs-claim-1
Namespace:     default
StorageClass:  efs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
               volume.kubernetes.io/storage-provisioner: efs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       efs-app-1
Events:
  Type     Reason                Age                From                                                                                      Message
  ----     ------                ----               ----                                                                                      -------
  Normal   ExternalProvisioning  13s (x3 over 35s)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator
  Normal   Provisioning          3s (x6 over 35s)   efs.csi.aws.com_efs-csi-controller-575b89d87d-vv8g7_cc810a13-232f-43e0-a9c1-e8de0a178b72  External provisioner is provisioning volume for claim "default/efs-claim-1"
  Warning  ProvisioningFailed    3s (x6 over 35s)   efs.csi.aws.com_efs-csi-controller-575b89d87d-vv8g7_cc810a13-232f-43e0-a9c1-e8de0a178b72  failed to provision volume with StorageClass "efs": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied

$ kubectl get pv                  
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                 STORAGECLASS   REASON   AGE
pvc-0a2b650d-9f2e-42b8-8f29-9b26783bbaa7   32Gi       RWO            Delete           Bound    kubecost/kubecost-cost-analyzer       gp3                     11m
pvc-9bd82ee9-b913-43a8-88c6-ae09ab9ba77d   32Gi       RWO            Delete           Bound    kubecost/kubecost-prometheus-server   gp3                     11m

Terminal Output Screenshot(s)

Additional context

kubernetes-sigs/aws-efs-csi-driver#709 (comment)

[simonsCatalyst]

Thanks @jamiemo for this, I found this issue and it helped solve my problem elsewhere.

The policy without elasticfilesystem:TagResource was working just fine for me until recently. Allowing elasticfilesystem:* just seemed to much :-)

[jamiemo]

elasticfilesystem:TagResource permissions have recently been added to the driver example, although tags appear to have been created by the driver for some time.