Downloading an artifact from a private s3 fails when directly specifying IAM keys

[agarfer1]

Nomad version

Nomad v1.7.6
BuildDate 2024-03-12T07:27:36Z
Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e

Operating system and Environment details

Linux 6.5.0-18-generic 18~22.04.1-Ubuntu x86_64

Issue

The documentation for the artifact states that this is one of the allowed formats when downloading from s3:

artifact {
  source = "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
}

This works correctly when the IAM credentials are read from the host, however when I specify those credentials directly, that bucket format doesn't work.

Reproduction steps

Trying to fetch an artifact like this:

artifact {
  source = "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
  options {
    aws_access_key_id     = "<id>"
    aws_access_key_secret = "<secret>"
  }
}

The error is similar to:

failed to download artifact "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz": getter subprocess failed: exit status 1: failed to download artifact: AccessDenied: Access Denied status code: 403, request id: XYZ, host id: w9ypewbKeo[......]4+Nep=

By just removing s3:// from the source, the object can be fetched now:

artifact {
  source = "my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
  options {
    aws_access_key_id     = "<id>"
    aws_access_key_secret = "<secret>"
  }
}

This works both when explicitly specifying the credentials and when they are fetched from the credentials file. Haven't tested this with ec2 instance profiles.

Credit to davidr912 in this comment for the solution.

Expected Result

The file should be downloaded just like when using the IAM credentials file.

Actual Result

The download fails.

[jrasell]

Hi @agarfer1 and thanks for raising this issue. Nomad imports the go-getter library, and therefore I think this issue should be moved to that repository as a feature enhancement.