Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add private S3 byte-range access to netcdf-3 #2117

Closed
DennisHeimbigner opened this issue Oct 1, 2021 · 1 comment
Closed

Add private S3 byte-range access to netcdf-3 #2117

DennisHeimbigner opened this issue Oct 1, 2021 · 1 comment

Comments

@DennisHeimbigner
Copy link
Collaborator

Currently, the netcdf-3 byte range driver cannot access private buckets in Amazon S3.
@DennisHeimbigner
Copy link
Collaborator Author

Resolved by upcoming PR

DennisHeimbigner added a commit to DennisHeimbigner/netcdf-c that referenced this issue Oct 30, 2021
@DennisHeimbigner
Fix additional S3 support issues
f6e25b6 
re: Unidata#2117
re: Unidata#2119

* Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case.
* The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows.
* Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails.
* Switch testing from stratus to the Unidata bucket on S3.
* Improve support for the s3: url protocol.
* Add a s3 specific utility code file: ds3util.c
* Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3.

## Misc.

* Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr.
* Add calls to nc_finalize in the utilities in case atexit is disabled.
* Add header only json parser to the distribution rather than as a built source.
@DennisHeimbigner DennisHeimbigner mentioned this issue Oct 30, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DennisHeimbigner