Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADD: GPM-API Tutorial #84

Merged
merged 4 commits into from
Aug 23, 2024
Merged

Conversation

ghiggi
Copy link
Contributor

@ghiggi ghiggi commented Aug 22, 2024

This PR adds a draft of the tutorial introducing GPM-API that I think can already be merged.

I plan to fix a couple of stuffs and make a new package release in the coming days and then open another PR to update the dependencies if you agree.

Copy link

github-actions bot commented Aug 22, 2024

👋 Thanks for opening this PR! The Cookbook will be automatically built with GitHub Actions. To see the status of your deployment, click below.
🔍 Git commit SHA: f895b02
✅ Deployment Preview URL: https://openradar.github.io/erad2024/_preview/84

@kmuehlbauer
Copy link
Collaborator

Thanks @ghiggi, I'll merge tomorrow, if everything runs well. Agreed, to add changes in subsequent PR's.

@ghiggi
Copy link
Contributor Author

ghiggi commented Aug 22, 2024

Hey @mgrover1 ! Could you please upload the (unzipped) sample GPM and xradar Zarr Stores on the Pythia bucket and provide me with the path to it?
You can download the zipped Zarr Stores here. Thanks 1000 !

@mgrover1
Copy link
Contributor

Absolutely I can @ghiggi 👍 I can take care of it tomorrow morning!

@kmuehlbauer
Copy link
Collaborator

@ghiggi There is an example how to grab the data from locally cached fsspec here (see also PyDDA Notebook):

# Set the URL and path for the cloud
URL = "https://js2.jetstream-cloud.org:8001/"
path = f"pythia/radar/erad2024"


fs = fsspec.filesystem("s3", anon=True, client_kwargs=dict(endpoint_url=URL))

fs.glob(f"{path}/*")

files = fs.glob("pythia/radar/erad2024/gpm-api/*")
local_files = [
    fsspec.open_local(
        f"simplecache::{URL}{i}", s3={"anon": True}, filecache={"cache_storage": "."}
    )
    for i in files
]

# grab the file with eg. `local_files[0]` 

or directly down load the data like this:

import os
import urllib.request
from pathlib import Path

# Set the URL for the cloud
URL = "https://js2.jetstream-cloud.org:8001/"
path = "pythia/radar/erad2024"
!mkdir -p data
files = [
    "gpm-api/2A.GPM.DPR.V9-20211125.20230820-S213941-E231213.053847.V07B.zarr))",
    "gpm-api/KNKX20230820_221341_V06.zarr",
]
for file in files:
    file_remote = os.path.join(path, file)
    file_local = os.path.join("data", Path(file).name)
    if not os.path.exists(file_local):
        print(f"downloading, {file_local}")
        urllib.request.urlretrieve(f"{URL}{file_remote}", file_local)

# files available in local data-subfolder

@kmuehlbauer
Copy link
Collaborator

Added the docker tag and going to merge when ready.

@kmuehlbauer kmuehlbauer merged commit 972ce00 into openradar:main Aug 23, 2024
2 checks passed
@kmuehlbauer
Copy link
Collaborator

Thanks @ghiggi!

github-actions bot pushed a commit that referenced this pull request Aug 23, 2024
@mgrover1
Copy link
Contributor

mgrover1 commented Aug 23, 2024

Hey @mgrover1 ! Could you please upload the (unzipped) sample GPM and xradar Zarr Stores on the Pythia bucket and provide me with the path to it? You can download the zipped Zarr Stores here. Thanks 1000 !

@ghiggi the datasets have been uploaded under

radar/erad2024/gpm_api

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants