[data] Add iterator batch_format=None support, which will yield batches in the current batch format with zero copies #33562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericl
requested review from
scv119,
clarkzinzow,
jjyao,
jianoaix and
c21
as code owners
amogkam
approved these changes
Mar 21, 2023
| @@ -379,7 +379,7 @@ def map_batches( | |||
| *, | |||
| batch_size: Optional[Union[int, Literal["default"]]] = "default", | |||
| compute: Optional[Union[str, ComputeStrategy]] = None, | |||
| batch_format: Literal["default", "pandas", "pyarrow", "numpy"] = "default", | |||
| batch_format: Optional[str] = "default", | |||
There was a problem hiding this comment.
keep it as Optional[Literal] for full explicitness of supported batch formats?
There was a problem hiding this comment.
I feel like that is hard to maintain (per the inconsistencies in the code already), so opted to go unify on the shorter signature.
|
lets also keep the documentation changes from https://github.com/ray-project/ray/pull/33536/files#diff-988f3832ac94d085daf61260175e2580920ebd1521dc760f58b426b94379d5b7L235? |
8 tasks
Signed-off-by: Eric Liang <ekhliang@gmail.com>
Done |
clarkzinzow
approved these changes
Mar 22, 2023
| @@ -540,7 +540,9 @@ def map_batches( | |||
| (promotes tables to Pandas and tensors to NumPy), ``"pandas"`` to select | |||
| ``pandas.DataFrame``, "pyarrow" to select ``pyarrow.Table``, or | |||
| ``"numpy"`` to select ``numpy.ndarray`` for tensor datasets and | |||
| ``Dict[str, numpy.ndarray]`` for tabular datasets. Default is "default". | |||
| ``Dict[str, numpy.ndarray]`` for tabular datasets, or None to return | |||
| the underlying block exactly as is with no additional formatting. | |||
There was a problem hiding this comment.
Nice, I like batch_size=None a good bit more than adding another literal string!
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: elliottower <elliot@elliottower.com>
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: Jack He <jackhe2345@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR is a cleanup of #33536
It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format.