HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused #6514
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adoroszlai
approved these changes
Apr 12, 2024
Comment on lines
624
to
628
| @Test | ||
| void testECReconstructionWithPartialStripe() | ||
| void testECReconstructParityWithPartialStripe() | ||
| throws Exception { | ||
| testECReconstructionCoordinator(ImmutableList.of(4, 5), 1); | ||
| } |
There was a problem hiding this comment.
nit: this can be removed, since recoverableMissingIndexes includes (4, 5), so testECReconstructionCoordinatorWithPartialStripe covers the same case.
It's OK to do it in #6515, so this can be merged as is.
There was a problem hiding this comment.
Good point. I will fix this when I rebase #6515
|
Thanks for the review @adoroszlai - merging this to master. |
Tejaskriya
pushed a commit
to Tejaskriya/ozone
that referenced
this pull request
Apr 17, 2024
sodonnel
added a commit
to sodonnel/hadoop-ozone
that referenced
this pull request
Apr 17, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85) Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestContainerCommandsEC.java
xichen01
pushed a commit
to xichen01/ozone
that referenced
this pull request
Apr 17, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85)
xichen01
pushed a commit
to xichen01/ozone
that referenced
this pull request
Apr 18, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85)
xichen01
pushed a commit
to xichen01/ozone
that referenced
this pull request
Apr 18, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85)
sodonnel
added a commit
to sodonnel/hadoop-ozone
that referenced
this pull request
Apr 18, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85) Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestContainerCommandsEC.java
jojochuang
pushed a commit
to jojochuang/ozone
that referenced
this pull request
May 29, 2024
…if it is unused (apache#6514) (cherry picked from commit cba8c85)
swamirishi
pushed a commit
to swamirishi/ozone
that referenced
this pull request
Jun 10, 2024
… data index if it is unused (apache#6514) (cherry picked from commit cba8c85) Change-Id: I45c8c731e7179272d14bde666cea18b5b99dfb7c
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Given a small EC block:
So that it is less than a full stripe and does not use all the datanodes.
When reconstruction happens to replace a replica which is not used in the stripe, the unused containers are not issued with the put block to store the details of the empty block within the container. Note that the container replica will likely have other blocks, so it will still get reconstructed, but it will not be given a reference to this empty block.
All containers are checked for the presence of all blocks during reconstruction. If any of the containers do not have a reference to the block, it is considered an orphan block / abandoned stripe and will not be reconstructed.
Therefore if one replica has no entry for the block, then it is used it another reconstruction for another replica later, that block will not get reconstructed into a second replica. Over time this can result in the reference getting removed from all copies.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10681
How was this patch tested?
Reproduced the issue using a new unit test, and then fixed the code to make the test pass.