[QST]!NOT_FOUND <InMemoryTableScanExec> cannot run on GPU #5398
-
|
What is your question?
Most of my operations were based on Spark SQL. Specially, dataframe.cache() were applied for in-memory acceleration. My question is that does this warning represent that it can not scan GPU in-memory data or datasets haven't been cached to GPU? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments
-
|
We are still working on optimizing cache and persist for the GPU see #11 Because of how the code works in spark it is going to require changes to spark to be able to even do things like optimizing the compression of the data using the GPU. Currently, this is being worked on as a part of apache/spark#29067. But even after those changes, it would not be enough to cache the data in GPU memory. We might be able to hack something together to make that work, but likely even further changes would be required to spark and the block manager to be able to do something like that. |
Beta Was this translation helpful? Give feedback.
-
|
@PerkzZheng is it Okay if I close this? |
Beta Was this translation helpful? Give feedback.
-
|
@revans2 Sure, thank you for your reply ! |
Beta Was this translation helpful? Give feedback.
We are still working on optimizing cache and persist for the GPU see #11
Because of how the code works in spark it is going to require changes to spark to be able to even do things like optimizing the compression of the data using the GPU. Currently, this is being worked on as a part of apache/spark#29067. But even after those changes, it would not be enough to cache the data in GPU memory. We might be able to hack something together to make that work, but likely even further changes would be required to spark and the block manager to be able to do something like that.