Device synchronize prior to freeing a set of RapidsBuffer #8936
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alessandro Bellina <abellina@nvidia.com>
This was referenced Aug 6, 2023
|
This succeeded 6 times when I ran it at 30TB. One of the runs had a task failure unrelated (not an illegal access) but scary as well #8939 |
|
build |
firestarman
reviewed
Aug 7, 2023
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsBufferCatalog.scala
Show resolved
Hide resolved
jlowe
previously approved these changes
Aug 7, 2023
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsBufferCatalog.scala
Outdated
Show resolved
Hide resolved
…Catalog.scala Fix comment as suggested Co-authored-by: Jason Lowe <jlowe@nvidia.com>
|
build |
revans2
approved these changes
Aug 7, 2023
jlowe
approved these changes
Aug 7, 2023
jlowe
changed the title
This was referenced Aug 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #8610
This issue changes a function in
RapidsBufferCatalog,spillBufferAndFree, to just copy the buffer to the appropriate tier, but the actual freeing part happens as the batch of copies have been already scheduled. What happens then is:safeFreeon the set of spilled buffers.The real solution would be to create an event per thread and synchronize at spill time against the event that was used to record for a specific rapids buffer.
I have run with this patch 6 times and no illegal access so far. I did have this failure which caused a task failure: #8939, that needs to be looked at. But this is not related to this change.
Follow on for Cuda.Event usage: #8937
Follow on for ColumnView issue in lazy spillable gather map #8938