✨ XbatcherSlicerIterDataPipe for slicing xarray.DataArray #22
+326
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Batch generation from xarray datasets! Currently using commit xarray-contrib/xbatcher@d3e1c2f
An interable-style DataPipe for creating chips from xarray.DataArray objects! Uses xbatcher to do the windowed slicing. Included a doctest and a unit test, added a new section in the API docs and an intersphinx mapping for xbatcher.
3 tasks
New extras group that includes `xbatcher`. Documented in CONTRIBUTING.md and updated GitHub Actions CI so that the full unit tests runs with `xbatcher` too.
Making sure that the climate science folks with their n-dimensional datasets can use XbatcherSlicer properly too!
weiji14
commented
Jul 5, 2022
Comment on lines
+103
to
+104
| # def __len__(self) -> int: | ||
| # return len(self.source_datapipe) |
There was a problem hiding this comment.
Note to future self, figure out whether there's a way to get the length of the datapipe lazily via https://github.com/pangeo-data/xbatcher/blob/d3e1c2f75dd0eea4e699b3398ba4d8bc1035a5e5/xbatcher/generators.py#L138-L139
This was referenced Jul 6, 2022
weiji14
added a commit
that referenced
this pull request
Aug 14, 2022
Left out on adding this if-statement in #22. So here's the patch!
weiji14
added a commit
that referenced
this pull request
Aug 14, 2022
Ensure that a helpful ModuleNotFoundError is raised when attempting to use XbatcherSlicer without xbatcher being installed. * 🩹 Raise ModuleNotFoundError when xbatcher not installed Left out on adding this if-statement in #22. So here's the patch! * ✏️ Fix type hints for xbatcher and datashader source_datapipe Need to use the full name xarray.DataArray instead of xr.DataArray to have the intersphinx link work.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An iterable-style DataPipe for creating chips from
xarray.DataArrayobjects! The windowed slicing is done using xbatcher.Note that
xbatcheris only used here to create slices of data (or chips/tiles), the actual creation of mini-batches (lumping together several slices/chips/tiles of images) will be handled by https://pytorch.org/data/0.4.0/generated/torchdata.datapipes.iter.Batcher.htmlPreview at https://zen3geo--22.org.readthedocs.build/en/19/api.html#module-zen3geo.datapipes.xbatcher
Note that since
xbatcheris made an optional dependency, users would need to dopip install zen3geo[raster]to install the extra 'raster' packages that includesxbatcher(and possibly more in the future).TODO:
xbatcheras optional dependencyXbatcherSlicerIterDataPipexbatcherdependency in full tests runxarray.Datasetobjects