This repository has been archived by the owner on Sep 30, 2024. It is now read-only.
Load diag data from s3 #104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
netCDF4 1.6 and above are required to work on Apple Silicon Macs. While we're at it, run poetry update to update the rest of our dependencies.
We want to be able to distinguish between s3 and local files so have decide to use URI's to indicate which storage backend to use. The S3 implementation is just a stub at this point but the file:// handling should work.
Closed
Installing Poetry via pipx before installing Python 3.9 was resulting in Poetry using the default version of Python (in this 3.8) instead of the required Python 3.9. This was causing errors when using newer language features.
This reverts commit 43a0c46. Turns out, we need poetry installed in order to use the cache when setting up Python in later steps.
Otherwise, Poetry uses the default Python 3.8 environment on the runner.
Each step runs in its own process in GitHub Actions so environment changes are not preserved between steps. I may need to add this to the `poetry run ...` commands as well. I hope it's unnecessary. Apparently Poetry has some "well-known" issues with correctly detecting the desired Python version. See: * Poetry issue: python-poetry/poetry#655 * setup-python workaround: actions/setup-python#374 (comment)
It sounds like Poetry will grab the Python env from the cache. The cached python is incorrect so hopefully this will result in the correct Python version. See: actions/setup-python#374 (comment)
I ended up needing to use fsspec's simple cache to download the files to local disk as xarray doesn't support reading netcdf directly from object storage - only zarr formatted datasets.
esheehan-gsl
suggested changes
Oct 25, 2022
| # and a file that's not there. It raises a ValueError even for missing | ||
| # files. We raise a FileNotFoundError to make debugging easier. | ||
| if not bucket.exists(diag_file): | ||
| print(f"No such file: '{str(diag_file)}'") |
There was a problem hiding this comment.
I’m pretty sure we have logging configured, so this would be better as
Suggested change
| print(f"No such file: '{str(diag_file)}'") | |
| current_app.logger.error(f"No such file: '{str(diag_file)}'") |
There was a problem hiding this comment.
Oh, that's a great catch. Thanks! I'll go ahead and switch those to logging statements.
Resolve conlicts with the poetry.lock file
We want the app in a demo-able state so we'll work on testing the S3 functionality in a new PR as it will require some more involved test setup.
|
Thanks, Evan. I've gone ahead and updated the bare This should let us run the demo from AWS instead of a local version of the site. I've been chatting with the VLab folks to set up permission for the API pod to access our S3 bucket in AWS. |
Minimum allowed line rate is |
Minimum allowed line rate is |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a protocol to the URIs in order to distinguish if the program should be looking on the local disk or on an S3 bucket for our data files.
Closes #80