Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Virtual-hosted bucket style doesn't work properly with AWS_ENDPOINT_URL #56

Open
craigds opened this issue Oct 14, 2024 · 0 comments

Comments

@craigds
Copy link

craigds commented Oct 14, 2024

Using a bucket name with no dots in it in conjunction with AWS_ENDPOINT_URL breaks the URL assembly that arbiter is doing:

export AWS_ENDPOINT_URL='https://localhost.localstack.cloud:4566/'

# echo '[{"type": "readers.copc", "filename": "s3://my-dev-bucket/file.laz"}]' | CURL_VERBOSE=1 pdal pipeline --stdin
Curl config:
	timeout: 5s
	followRedirect: true
	verifyPeer: true
	caBundle: (default)
	caInfo: /opt/certifi/cacert.pem
	Proxy: (default)
* Could not resolve host: my-dev-bucket.https
* Closing connection
Curl failure: Couldn't resolve host name
^C

It seems to be assuming the endpoint url starts with a domain name rather than a scheme? so the result is probably something like https://my-dev-bucket.https://localhost.localstack.cloud:4566/file.laz

This seems fixed as long as the bucket name has at least one . in it, which forces arbiter not to use virtual-hosted style URLs.

Possible fixes could be any of:

  1. disable virtual hosted always (what's the upside? there seems to be a vague promise from AWS that non-virtual-hosted will go away eventually but they haven't given an actual date yet, and there's no proposed options for dot-named buckets)
  2. make it disableable via an env var (GDAL uses AWS_VIRTUAL_HOSTING=FALSE for this purpose)
  3. disable virtual hosting automatically when AWS_ENDPOINT_URL is in use
  4. fix the broken url assembly so it preserves the scheme.

(4) by itself would still be a pain because I'd have to add a subdomain alias for my docker container, but it's workable.

My suggestion would be to implement both (2) and (4) to fix this.

craigds added a commit to craigds/arbiter that referenced this issue Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant