/vsis3/ unable to use IAM roles for service accounts (EKS, Kubernetes)

[scottyhq]

Expected behavior and actual behavior.

A common setup for applications running on AWS EKS is to use IAM roles for service accounts. This way users don't have access to EC2 metadata, but are able to have IAM permissions associated with their running application. Currently, the GDAL tries searching for credentials in an ordered way using /vsis3/, but doesn't know how to get these credentials (search for "exchanging the Kubernetes-issued OIDC token for AWS role credentials" in this blog post for a technical explanation of how the credentialing works).

Steps to reproduce the problem.

It's hard to share a reproducible environment, but I think the following captures the issue:

1. A user running on such a system has credentials as can be seen running aws configure list

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************FR4R assume-role-with-web-identity    
secret_key     ****************3A9O assume-role-with-web-identity    
    region                us-west-2              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

2. Using the AWS CLI it's possible to read from permitted S3 buckets: aws s3 ls s3://snowex-data/asf_vertex.png

                           PRE tutorial-data/
2021-07-02 16:40:58    1788203 asf_vertex.png

3. GDAL tries to find local credential files (nonexistent), and ultimately tries directly for EC2 Instance metadata (HTTP: Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials which is typically blocked on K8s applications), and finally times out.
   CPL_DEBUG=ON CPL_CURL_VERBOSE=YES gdalinfo /vsis3/snowex-data/asf_vertex.png

HTTP: Fetch(http://169.254.169.254/latest/api/token)
HTTP: libcurl/7.77.0 OpenSSL/1.1.1k zlib/1.2.11 libssh2/1.9.0 nghttp2/1.43.0
HTTP: GDAL was built against curl 7.71.1, but is running against 7.77.0.
HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10
* Couldn't find host 169.254.169.254 in the (nil) file; using defaults
*   Trying 169.254.169.254:80...
* Connection timed out after 1000 milliseconds
* Closing connection 0
gdalinfo failed - unable to open '/vsis3/snowex-data/asf_vertex.png'

Operating system

Ubuntu 20.04 running on a AWS EKS k8s 1.19 cluster

GDAL version and provenance

GDAL 3.2.1, released 2020/12/29 from conda-forge

[metasim]

Just discovered this problem also. Having to attach policy to node group to get it to work, which is not very secure.

[nsshah1288]

Hi, I am using AWS EKS IAM roles for service accounts, and I am running into a similar issue. gdal version is 3.5.1

I saw that a bunch of commits were made to address this, but is there a gdal version that contains fixes for this? @rouault @ashangit @g8sqh

Error I am getting is
RasterioIOError: Read or write failed. '/vsis3/our-private-bucket/some-file.tif' not recognized as a supported file format.

I am able to read the same file in EKS using the AWS CLI fine (with aws s3 cp), but when trying to read the file with rasterio, it fails with the above error

Thanks!

[rouault]

implemented for upcoming 3.6.0 (nov 2022) per #6074

[nsshah1288]

Ok, thank you for the response. Maybe update the documentation here https://gdal.org/user/virtual_file_systems.html#:~:text=If%20AWS_ROLE_ARN%20and%20AWS_WEB_IDENTITY_TOKEN_FILE%20are%20defined%20we%20will%20rely%20on%20credentials%20mechanism%20for%20web%20identity%20token%20based%20AWS%20sts%20action%20AssumeRoleWithWebIdentity%20(See.%3A%20https%3A//docs.aws.amazon.com/eks/latest/userguide/iam%2Droles%2Dfor%2Dservice%2Daccounts.html) to say that this is an upcoming feature that's not yet implemented?

[rouault]

Maybe update the documentation here

done in 05d0a0c