Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data] [Docs] Docs for working with PyTorch #36880

Merged
merged 18 commits into from
Jun 30, 2023

Conversation

amogkam
Copy link
Contributor

@amogkam amogkam commented Jun 27, 2023

Adds a guide for how PyTorch users can use Ray Data. This includes a migration guide from Torch Dataset/DataLoader to Ray Data. Closes #31418

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

amogkam added 6 commits June 24, 2023 19:17
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
amogkam added 5 commits June 27, 2023 14:32
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
@amogkam amogkam requested a review from a team as a code owner June 28, 2023 17:54
@@ -34,6 +34,7 @@ tensorflow; sys_platform != 'darwin' or platform_machine != 'arm64'
tensorflow-macos; sys_platform == 'darwin' and platform_machine == 'arm64'
tensorflow-datasets
torch
torchtext
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to requirements_test.txt. (Previous guidance was to use requirement-doc.txt. Going forward, we want requirements-doc.txt to be the minimum set of dependencies to build the documentation)

doc/source/data/api/from_other_data_libs.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
doc/source/data/working-with-pytorch.rst Outdated Show resolved Hide resolved
amogkam added 2 commits June 28, 2023 14:51
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
@amogkam amogkam requested a review from bveeramani June 28, 2023 22:20
Copy link
Member

@bveeramani bveeramani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for the saving datasets section. If we can write to formats other than Parquet and NumPy, it's good as is. Otherwise, let's update it

Signed-off-by: amogkam <amogkamsetty@yahoo.com>
amogkam added 2 commits June 29, 2023 14:30
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
…-pytorch

Signed-off-by: amogkam <amogkamsetty@yahoo.com>
amogkam added 2 commits June 29, 2023 16:03
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Copy link
Contributor

@matthewdeng matthewdeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stamp

@amogkam amogkam merged commit 80e458c into ray-project:master Jun 30, 2023
@amogkam amogkam deleted the working-with-pytorch branch June 30, 2023 03:30
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
Adds a guide for how PyTorch users can use Ray Data. This includes a migration guide from Torch Dataset/DataLoader to Ray Data. Closes ray-project#31418

---------

Signed-off-by: amogkam <amogkamsetty@yahoo.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[AIR] [Data] Guide to move from DataLoader to Ray Data
5 participants