Control artifact go-getter "mode"

[jippi]

Nomad version

Nomad v0.5.6

Issue

When using go-getter within s3, to download a specific file, deeply nested inside a directory hierarchy, the current usage of go-getter requires some unnecessary wide s3 policies to work.

With the current usage, you need a policy similar to the one below to make downloading artficats work, since go-getter run in any Mode by default - meaning it will try to traverse the bucket to find the file, and other files (if it happen to be a directory)

Being able to control the Mode would allow us to drop the ListObjects policy from our aws instances, and further harden our security footprint.

Looking at the code in go-getter, it seem like it tries, by default, to be recursive and download multiple files in the bucket if the source in nomad artifact happened to be a folder rather than a file

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket/folder1/folder2/folder3/file"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:ListObjects"
            ],
            "Resource": [
                "arn:aws:s3:::bucket/*",
                "arn:aws:s3:::bucket"
            ],
            "Effect": "Allow"
        }
    ]
}

Reproduction steps

any mode (current in nomad)

Without ListObjects policy:

go-getter -mode any s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 Error downloading: AccessDenied: Access Denied

WIth ListObjects policy:

go-getter -mode any s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 OK!

dir mode

Without ListObjects policy:

go-getter -mode dir s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 Error downloading: AccessDenied: Access Denied

WIth ListObjects policy:

go-getter -mode dir s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 OK!

file mode

Without ListObjects policy:

go-getter -mode file s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 OK

With ListObjects policy:

go-getter -mode file s3::https://s3.amazonaws.com/bucket/folder1/folder2/folder3/file file
2017/05/30 10:16:06 OK

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.