You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This concerns the shuffle cleanup fix introduced in SPARK-46957. Per description:
This is a long-standing bug in decommission where the migrated shuffle files can't be cleaned up from the executor. Normally, the shuffle files are tracked by taskIdMapsForShuffle during the map task execution. Upon receiving the RemoveShuffle(shuffleId) request from driver, executor can clean up those shuffle files by searching taskIdMapsForShuffle. However, for the migrated shuffle files by decommission, they lose the track in the destination executor's taskIdMapsForShuffle and can't be deleted as a result.
This might need evaluation by someone familiar with Spark Shuffle. It appears to be a missed condition where shuffle files might not be cleaned up.
The text was updated successfully, but these errors were encountered:
This concerns the shuffle cleanup fix introduced in SPARK-46957. Per description:
This might need evaluation by someone familiar with Spark Shuffle. It appears to be a missed condition where shuffle files might not be cleaned up.
The text was updated successfully, but these errors were encountered: