Pipe: stop pipe using restarting strategy to unpin the wal's reference count to avoid WAL stacking #11971
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/PipeTaskAgent.java
Show resolved
Hide resolved
SteveYurongSu
approved these changes
Jan 30, 2024
SteveYurongSu
changed the title
Caideyipi
added a commit
to Caideyipi/iotdb
that referenced
this pull request
Feb 6, 2024
…reference count to avoid WAL stacking (apache#11971)" This reverts commit d0928eb.
SzyWilliam
pushed a commit
to SzyWilliam/iotdb
that referenced
this pull request
Nov 26, 2024
…e count to avoid WAL stacking (apache#11971)
SzyWilliam
pushed a commit
to SzyWilliam/iotdb
that referenced
this pull request
Nov 26, 2024
…wal's reference count to avoid WAL stacking (apache#11971)" to avoid unnecessary pipe drop during subtask exception handling (apache#12031) This reverts commit d0928eb.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Change the execution flow of the
stop pipe:When the pipe is dropped and re-created with the same pipeMeta, it will re-trigger the collection of historical data, and most of the files it pins are tsfile files not wal, because the wal files are flushed. By taking advantage of this property, we can make sure that the subsequent pins are new files, not old wal files, thus avoiding the wal stacking.