Skip to content

Commit

Permalink
🎨 Add figure showing xbatcher's features for slicing n-dim arrays (#8)
Browse files Browse the repository at this point in the history
Illustration of xbatcher's features of slicing multi-dimensional arrays on-the-fly with named variables, useful for e.g. time-series or multi-variate oceanography / climate model outputs. Diagram on the left shows how a datacube is sliced, and code block on the right shows how regular slicing code compares to xbatcher.

At the bottom, more features are mentioned such as lazy loading, xarray acessors, and the experiment cache mechanism. Also mentioned upcoming roadmap like shuffling/sampling utilities and async loading of batches.
  • Loading branch information
weiji14 authored Oct 15, 2023
1 parent cc76120 commit de7fd79
Show file tree
Hide file tree
Showing 3 changed files with 5,256 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ about the challenges and ideas on scaling machine learning in the geosciences
with the [Pangeo ML Working Group](https://pangeo.io/meeting-notes.html#working-group-meetings).

[![NVIDIA GPUDirect Storage schematic](https://github.com/weiji14/foss4g2023oceania/assets/23487320/b339605a-492b-4c5c-8a06-b14ebcd258b7)](https://github.com/weiji14/foss4g2023oceania/pull/6)
[![xbatcher n-dimensional slicing](https://github.com/weiji14/foss4g2023oceania/assets/23487320/38af26ca-83ea-4798-be7f-95608ad56df0)](https://github.com/weiji14/foss4g2023oceania/pull/8)

# Getting started

Expand Down
1 change: 1 addition & 0 deletions figures/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Folder to store all the figures used in the presentation.

- [Pangeo ML Ecosystem 2023 diagram](foss4g2023oceania_pangeo_ml_talk.excalidraw) | https://excalidraw.com/#json=Z0T6mFgNZnFSjN3rs349O,YdPkNNwey7rS5OtWgvSkyQ
- [NVIDIA GPU Direct Storage](nvidia_gpu_direct_storage.excalidraw) | https://excalidraw.com/#json=PBEuODqMxdJFVxGrgLzkP,ZAGNmAF2OJmxeq_nIDbPEA
- [xbatcher n-dimensional slicing](xbatcher_ndim_slicing.excalidraw) | https://excalidraw.com/#json=4iuB5hMt9ihbLNrN8Pirw,Ktfz9P8oHqQS5IJrBzoC8A

## Logos

Expand Down
Loading

0 comments on commit de7fd79

Please sign in to comment.