🚸 Walkthrough on creating batches of data chips #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initial draft tutorial on creating batches of chipped data from full-size satellite scenes! Will be working with Sentinel-1 GRD GeoTIFFs, let's see how far this will go.
Bump minimum pyogrio version from 0.4.0a1 to 0.4.0 and include new XbatcherSlicer feature! Also get the refactored RioXarrayReader in.
Walkthrough how to cut up a large satellite scene into multiple smaller chips of size 512 pixels by 512 pixels. Heavy lifting done by xbatcher which handles slicing along dimensions and overlapping strides. Needed a hacky workaround in XbatcherSlicer to fix a ValueError due to the xarray.DataArray name not being set (though it should be).
Fix readthedocs build failure because xbatcher was not installed.
Finalize tutorial by converting chips from xarray.Dataset to torch.Tensor and stacking them per mini-batch! Debated on whether to have the xarray collate function in the codebase, but let's wait for updates on xbatcher's end (xarray-contrib/xbatcher#71). Also renamed the tutorial file from batching to chipping and added more emojis to the intro section.
weiji14
changed the title
weiji14
added a commit
that referenced
this pull request
Sep 7, 2022
Just a random collection of mostly documentation-related patches. Patches type-hints in #52, isort imports in #35, mention functional name of IterDataPipe in walkthroughs #8 and #20, and remove mention of returned tuple to patch #33. * 🏷️ Add specific type hints for mask_datapipe in geopandas.py Should be either an xarray.DataArray or xarray.Dataset. * 🚨 Sort spatialpandas imports in datashader.py Ran isort to sort spatialpandas.geometry imports alphabetically. Also intersphinx linked the `.crs` attribute to geopandas.GeoDataFrame.crs. * 💬 Mention functional name of IterDataPipe in walkthroughs So people don't get confused on why the class-form like `Collator` is mentioned but `.collate` was used instead. * 📝 Remove mention of tuple being returned in test_pyogrio_reader Forgot to edit the unit test's docstring. Patches #33. * 🍻 It's GeoPackage and GeoDataFrame, not GeoTIFF and DataArray Need to be more careful when copying and pasting stuff.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial tutorial on creating batches of chipped data (256x256) from full-size satellite scenes (10000x10000) with different coordinate reference systems! Using example Sentinel-1 GRD GeoTIFFs over Osaka and Tokyo in Japan.
Preview at https://zen3geo--20.org.readthedocs.build/en/20/chipping.html
https://planetarycomputer.microsoft.com/explore?c=137.1529%2C35.0944&z=7.94&v=2&d=sentinel-1-grd&s=false%3A%3A100%3A%3Atrue&ae=0&m=cql%3A08211c0dd907a5066c41422c75629d5f&r=VV%2C+VH+False-color+composite
TODO: