Pull based native execution #70
Labels
enhancement
New feature or request
Comments
himadripal
pushed a commit
to himadripal/datafusion-comet
that referenced
this issue
Sep 7, 2024
* build: Move installation of cargo2junit to builder Install nigthly and use that to install cargo2junit. This avoids constant installations for each test run, as well as moving the details about the required toolchain for cargo2junit out of the build scripts. * build: Switch to comet-builder Originally the comet-builder would fail because of XML comments in one of the toml files. This has been addressed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the problem the feature request solves?
Comet native execution's scan is not started from native but from JVM. Thus Comet scan is push-based instead of pull-based. Although we pull next input batches from child operator in JVM, this new input is not pulled from native but pushed from JVM side.
For an operator like Expand, one input batch can produces multiple output batches. So we cannot pull next batch directly and push into native without peeking it. We need to "peek" into native side and see if any more output batch there. If so, we take it as next output, if not, we pull next input batch and push into native side to execute on it.
If we pull next input from child operator and push it into native without peek, new input will be ignored.
Not only we cannot have consistent way to get input for native operators. The code of input/output to native execution is harder to understand because we mix push-based and pull-based processing modes. This patch tries to make native execution fully pull-based.
Describe the potential solution
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: