-
-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ XbatcherSlicerIterDataPipe for slicing xarray.DataArray #22
Conversation
Batch generation from xarray datasets! Currently using commit xarray-contrib/xbatcher@d3e1c2f
An interable-style DataPipe for creating chips from xarray.DataArray objects! Uses xbatcher to do the windowed slicing. Included a doctest and a unit test, added a new section in the API docs and an intersphinx mapping for xbatcher.
New extras group that includes `xbatcher`. Documented in CONTRIBUTING.md and updated GitHub Actions CI so that the full unit tests runs with `xbatcher` too.
Making sure that the climate science folks with their n-dimensional datasets can use XbatcherSlicer properly too!
# def __len__(self) -> int: | ||
# return len(self.source_datapipe) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to future self, figure out whether there's a way to get the length of the datapipe lazily via https://github.com/pangeo-data/xbatcher/blob/d3e1c2f75dd0eea4e699b3398ba4d8bc1035a5e5/xbatcher/generators.py#L138-L139
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in #75!
Left out on adding this if-statement in #22. So here's the patch!
Ensure that a helpful ModuleNotFoundError is raised when attempting to use XbatcherSlicer without xbatcher being installed. * 🩹 Raise ModuleNotFoundError when xbatcher not installed Left out on adding this if-statement in #22. So here's the patch! * ✏️ Fix type hints for xbatcher and datashader source_datapipe Need to use the full name xarray.DataArray instead of xr.DataArray to have the intersphinx link work.
An iterable-style DataPipe for creating chips from
xarray.DataArray
objects! The windowed slicing is done using xbatcher.Note that
xbatcher
is only used here to create slices of data (or chips/tiles), the actual creation of mini-batches (lumping together several slices/chips/tiles of images) will be handled by https://pytorch.org/data/0.4.0/generated/torchdata.datapipes.iter.Batcher.htmlPreview at https://zen3geo--22.org.readthedocs.build/en/19/api.html#module-zen3geo.datapipes.xbatcher
Note that since
xbatcher
is made an optional dependency, users would need to dopip install zen3geo[raster]
to install the extra 'raster' packages that includesxbatcher
(and possibly more in the future).TODO:
xbatcher
as optional dependencyXbatcherSlicerIterDataPipe
xbatcher
dependency in full tests runxarray.Dataset
objects