Unable to provision volumes in iso-us-east-1 AWS region

[hazmat345]

/kind bug

What happened?
Attempting to provision a volume fails because the provisioner does not trust the AWS TLS certificate. The following event is generated:

failed to provision volume with StorageClass "efs": rpc error: code = Internal desc = Failed to fetch Access Points or Describe File System: List Access Points failed: RequestError: send request failed
caused by: Get "https://elasticfilesystem.us-iso-east-1.<rest of url>: tls: failed to verify certificate: x509: certificate signed by unknown authority

What you expected to happen?
The volume to be provisioned successfully.

How to reproduce it (as minimally and precisely as possible)?
Attempt to provision a volume in an AWS region that does not use a certificate issued by an authority in the standard CA certificate bundle.

Anything else we need to know?:
The certificate used by the AWS endpoint in my environment is not part of the standard CA bundle, so I need to mount my own CA trust bundle into the pods.

I suspect this would not longer be an issue if this PR is accepted: #1165

Environment

• Kubernetes version (use kubectl version): 1.28.8+rke2r1
• Driver version: 2.0.2
• Helm Chart version: 3.0.3

Please also attach debug logs to help us better diagnose

• Instructions to gather debug logs can be found here

[slambrose]

+1

[seanzatzdev-amazon]

Thank you for bringing this to our attention. We are investigating this issue. In the meantime, can you follow the log-collection steps linked here to provide us with debug logs?

[hazmat345]

Getting logs from my actual environment is going to be tricky... let me see if I can figure out a way to reproduce things.

[whoix]

@hazmat345 @slambrose

Is there a reason why you can't use the managed add-on for EFS with your EKS cluster? The add-on is supported in ISO regions and fully compatible. The managed add-on already auto injects the custom CA bundle used in ISO regions.

[slambrose]

@whoix For unfortunate "reasons", we are unable to use EKS at the moment, so we are building our RKE2 clusters on EC2 instances w/ terraform and ansible. I believe the PR was merged yesterday, so this issue can probably be closed now.