✨ XpySTACAssetReader for reading COG, NetCDF & Zarr STAC assets

[weiji14]

An iterable-style DataPipe for STAC asset data like Cloud-Optimized GeoTIFFs, NetCDF and Zarr!

Preview at https://zen3geo--87.org.readthedocs.build/en/87/api.html#zen3geo.datapipes.XpySTACAssetReader

Usage

from torchdata.datapipes.iter import IterableWrapper
from zen3geo.datapipes import XpySTACAssetReader

# Read in STAC Asset using DataPipe
collection_url: str = "https://planetarycomputer.microsoft.com/api/stac/v1/collections/nasa-nex-gddp-cmip6"
asset: pystac.Asset = pystac.Collection.from_file(href=collection_url).assets[
    "ACCESS-CM2.historical"
]
dp = IterableWrapper(iterable=[asset])
dp_xpystac = dp.read_from_xpystac()

# Loop or iterate over the DataPipe stream
it = iter(dp_xpystac)
dataset = next(it)

print(dataset.sizes)
# Frozen({'time': 23741, 'lat': 600, 'lon': 1440})

print(dataset.data_vars)
# Data variables:
#     hurs     (time, lat, lon) float32 ...
#     huss     (time, lat, lon) float32 ...
#     pr       (time, lat, lon) float32 ...
#     rlds     (time, lat, lon) float32 ...
#     rsds     (time, lat, lon) float32 ...
#     sfcWind  (time, lat, lon) float32 ...
#     tas      (time, lat, lon) float32 ...
#     tasmax   (time, lat, lon) float32 ...
#     tasmin   (time, lat, lon) float32 ...

print(dataset.attrs)
# {'Conventions': 'CF-1.7',
#  'activity': 'NEX-GDDP-CMIP6',
#  'cmip6_institution_id': 'CSIRO-ARCCSS',
#  'cmip6_license': 'CC-BY-SA 4.0',
#  'cmip6_source_id': 'ACCESS-CM2',
#  ...
#  'history': '2021-10-04T13:59:21.654137+00:00: install global attributes',
#  'institution': 'NASA Earth Exchange, NASA Ames Research Center, ...
#  'product': 'output',
#  'realm': 'atmos',
#  'references': 'BCSD method: Thrasher et al., 2012, ...
#  'resolution_id': '0.25 degree',
#  'scenario': 'historical',
#  'source': 'BCSD',
#  'title': 'ACCESS-CM2, r1i1p1f1, historical, global downscaled CMIP6 ...
#  'tracking_id': '16d27564-470f-41ea-8077-f4cc3efa5bfe',
#  'variant_label': 'r1i1p1f1',
#  'version': '1.0'}

TODO:

• Initial implementation with a doctest
• Add unit tests for both COG and Zarr STAC assets
• Think about the upper/lower casing of letters in XpySTACAssetReader
• etc

Personally, I've been debating whether to add xr.open_dataset and/or xr.open_zarr to zen3geo for months because:

1. While xarray is widely used by 'geo' folks, it is not just for geo, so it didn't seem proper to wrap a torchdata DataPipe for xarray here.
2. The Zarr format is also not only for geo.
3. There are discussions on deprecating open_zarr at Deprecate open_zarr in favor of open_dataset(..., engine='zarr') pydata/xarray#7495. Didn't want to wrap open_zarr only to have to drop it later.

These concerns become moot with the availability of xr.open_dataset(..., engine="stac") enabled by xpystac! It provides a single entrypoint to Zarr, Cloud-Optimized GeoTIFFs, and potentially more STAC Asset based datasets, and since STAC is spatiotemporal (read: geo), this fits naturally within zen3geo 😄

References:

• https://github.com/jsignell/xpystac/blob/0.0.1/xpystac/core.py
• Convenience methods for converting STAC objects / linked assets to data containers stac-utils/pystac#846

[weiji14]

Debating on whether to put xpystac under the 'raster' or 'stac' extras 🤔

[weiji14]

Decided to go with xpystac under the 'stac' extras for now (done at commit 49c29b1) since xpystac depends on pystac, xarray, and somewhat optionally on stackstac, so the Venn diagram overlaps more with the STAC ecosystem than raster-like stuff like xarray and Zarr.