Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow providing a service account key directly for GCS #3489

Merged
merged 3 commits into from
Jan 9, 2023

Conversation

scsmithr
Copy link
Contributor

@scsmithr scsmithr commented Jan 7, 2023

Which issue does this PR close?

Closes #3488

Rationale for this change

Use case:

We're storing service accounts keys external to where the object store client is
being created. We do not want to have to write the key to a file before creating
the object store client. This change allows for providing the key directly.

What changes are included in this PR?

Adds an appropriate method to the GCS object store builder for supplying the service account key directly. Only one of service account path or service account key may be provided, otherwise build will return an appropriate error.

Are there any user-facing changes?

An additional method on GCS object store builder.

There are currently no breaking changes, however I believe the ServiceAccount variant for the GoogleConfigKey should be renamed to ServiceAccountPath to better represent what that option is for. I held off on making that change because I saw that the changelog was already generated for 0.5.3 which includes the new GoogleConfigKey stuff, making that a breaking change. If that's an acceptable breaking change, I'm down to go ahead and do that in this PR as well.

Use case:

We're storing service accounts keys external to where the object store client is
being created. We do not want to have to write the key to a file before creating
the object store client. This change allows for providing the key directly.
@github-actions github-actions bot added the object-store Object Store Interface label Jan 7, 2023
@tustvold
Copy link
Contributor

tustvold commented Jan 8, 2023

Thank you, makes sense to me.

I believe the ServiceAccount variant for the GoogleConfigKey should be renamed to ServiceAccountPath to better represent what that option is for.

Yeah, probably not worth introducing a breaking change over, but perhaps we could deserialize "service_account_path" and "google_service_account_path" to allow for users to opt-in to using a more explicit config key. What do you think?

"google_service_account_path" and "service_account_path" can now be used.
@scsmithr
Copy link
Contributor Author

scsmithr commented Jan 8, 2023

perhaps we could deserialize "service_account_path" and "google_service_account_path" to allow for users to opt-in to using a more explicit config key. What do you think?

Seems reasonable to me. Latest commit has those included.

@tustvold tustvold merged commit eae993f into apache:master Jan 9, 2023
@ursabot
Copy link

ursabot commented Jan 9, 2023

Benchmark runs are scheduled for baseline = c746658 and contender = eae993f. eae993f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow providing service account key directly when building GCP object store client
3 participants