Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EXPORT S3]: Support Non-standard AWS S3 endpoints #8940

Closed
nswamy opened this issue Aug 8, 2023 · 0 comments · Fixed by #8978
Closed

[EXPORT S3]: Support Non-standard AWS S3 endpoints #8940

nswamy opened this issue Aug 8, 2023 · 0 comments · Fixed by #8978
Labels
kind/feature Something completely new we should consider.

Comments

@nswamy
Copy link

nswamy commented Aug 8, 2023

Use case

Users who have S3 in non-standard AWS endpoints (gov-cloud or private AWS cloud) cannot export their data to S3. We need to update export code to support non-standard aws endpoints, we could probably do a HEAD call on the s3 endpoint to validate the existence of s3 bucket.

return nil, errors.Errorf("Invalid S3 endpoint %q", uri.Host)

The general AWS S3 bucket URL is s3://s3.<region>.amazonaws.com/<bucket-name> however for classified regions and when using custom endpoints the URL would be s3://s3.<region>.<endpoint>/<bucket-name>
more info on private endpoint here

@nswamy nswamy added the kind/feature Something completely new we should consider. label Aug 8, 2023
mangalaman93 pushed a commit that referenced this issue Aug 26, 2023
Description: Many blob storage providers support the S3 API, e.g.:
Cloudflare R2, Backblaze B2. This fix allows to use any S3 endpoint.
Fixes: #8940
shivaji-kharse pushed a commit that referenced this issue Mar 12, 2024
Description: Many blob storage providers support the S3 API, e.g.:
Cloudflare R2, Backblaze B2. This fix allows to use any S3 endpoint.
Fixes: #8940
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Something completely new we should consider.
Development

Successfully merging a pull request may close this issue.

1 participant