Add experimental AWS_PROFILE support (#2178) #2891
Conversation
|
I agree with aws-vault being a better solution to store credentials. (With sso profiles there are no credentials stored in plain text files though). Having to type aws-vault exec --profile xxx application is just not very ergonomic in the case of short-lived/cli apps. |
It's been a while since I used it, but if memory serves it caches both the final credentials, and the returned SAML assertion. The former typically has a TTL of 1 hour, and the latter 24 hours. Certainly better than non-ephemeral credentials, but still not great 😅
FWIW you can type |
|
I have tested this locally and it at least appears to work for me, I therefore think it is ready for review. Perhaps @wjones127, @timvw and/or @roeap might be able to test this out / help review this? |
|
Works for me (tested on commit (2079a67)). Thank you! cargo test --package object_store --features aws_profile --lib aws::tests::s3_with_sso_profile -- --exact #[tokio::test]
async fn s3_with_sso_profile() -> Result<()>{
env::set_var("AWS_PROFILE", "XXX");
let s3 = AmazonS3Builder::from_env()
.with_bucket_name("YYY")
.build()?;
let path = Path::parse("ZZZ")?;
let files =
s3.list(Some(&path))
.await?
.try_collect::<Vec<ObjectMeta>>()
.await?;
assert!(files.len() > 0);
Ok(())
}
` |
|
Also verified that when AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_PROFILE are set that the first two get preference over AWS_PROFILE (similar to how aws cli behaves). |
|
Do we really need the full SDK for this? Skimming through https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html it seems this is only an ini file at a fixed (OS-specific) location where the section is the profile name and the key-value dict in that section are the env. variables we're looking for. Ah, I see:
|
|
Yeah, its a bit of a hack. Hopefully over time we can remove the need for it |
|
Benchmark runs are scheduled for baseline = 4620abf and contender = 9c315ce. 9c315ce is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
Are there plans to release a newer version of the object_store crate with these changes? |
|
I was waiting for a critical mass of features to warrant a release. I'll probably look to cut a release regardless in the next week or two. |
# Description - Add the support of `AWS_PROFILE` environment variable for AWS S3 - Bump version of `object_store` to `0.5.2` # Related Issue(s) - relate to apache/arrow-rs#3229 - relate to apache/arrow-rs#2891 - closes #855
Which issue does this PR close?
Closes #2178
Rationale for this change
Support for the more esoteric AWS auth options has been requested in various places, rather than forcing downstreams to workaround this by using an SDK such as aws-sdk-rust, lets just provide an option to use aws-sdk-rust to get credentials from a profile. This lets people explicitly opt-in to the full suite of credential providers, with the accompanying dependency explosion, whilst still using the same code for actual communication with object storage. I still would argue aws-vault is a superior solution for credential management than relying on inconsistent SDK implementations which store credentials in plain text on disk, but we should meet users where they currently are.
What changes are included in this PR?
Adds an optional aws_profile feature which uses aws-config to provide the full suite of AWS credential functionality, including SSO, external credential providers, role chaining, etc...
Are there any user-facing changes?
No