-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add enforce_concatenated_form
#2860
Conversation
248444a
to
99f61b9
Compare
Codecov Report
Additional details and impacted files
|
26d8225
to
0108f07
Compare
87db34e
to
ea53fe2
Compare
ea53fe2
to
99f61b9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! I noticed all of the typetracer tests that were added to tests/test_0449_merge_many_arrays_in_one_pass.py, too.
We talked about this in our meeting, and the code looks good to me, so this is ready to merge.
Co-authored-by: Jim Pivarski <jpivarski@users.noreply.github.com>
Tip
This PR makes the following changes:
_mergemany
no longer enforces a backendak.operations.ak_concatenate.enforce_concatenated_form
for enforcing a concatenated form.dask-awkward
needs to be able to compute a concatenation operation but without actually concatenating buffers. @jpivarski and I discussed the various ways we could go about this:enforce_type
to produce partitions of the correct type.enforce_form
to produce partitions of the correct form._mergemany
that doesn't touch buffers if they have known length-zero.Each of these solutions has drawbacks. In particular, modifying existing code to satisfy two constraints (perform an optimal merge in convenient amounts of code, and have a length-zero path) is more difficult than having a dedicated function that only handles the "don't touch buffers" case. As such, this PR instead adds a new function that acts like
enforce_form
but under the conditions that theform
was build fromak.concatenate((layout, ...), axis=0)
wherelayout
is the content being enforced toform
. This allows us to make many simplifying assumptions according to the implementation of_mergemany
.