Fixing failing compaction/parallel index jobs during upgrade due to new actions being available on the overlord. #15430
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ew actions not available on the overlord.
cryptoe
requested review from
kfaraz,
abhishekagarwal87 and
AmatyaAvadhanula
kfaraz
approved these changes
Nov 25, 2023
| @@ -67,16 +68,16 @@ public class RetrieveSegmentsToReplaceAction implements TaskAction<Collection<Da | |||
| private final String dataSource; | |||
|
|
|||
| @JsonIgnore | |||
There was a problem hiding this comment.
I think I had missed this earlier when this class was created. Not sure why we need @JsonIgnore here.
| @@ -84,19 +84,19 @@ default Collection<DataSegment> retrieveUsedSegmentsForInterval( | |||
| /** | |||
| * | |||
| * Retrieve all published segments which are marked as used and the created_date of these segments belonging to the | |||
| * given data source and interval from the metadata store. | |||
| * given data source and List<Interval> from the metadata store. | |||
There was a problem hiding this comment.
Nit: This doesn't render correctly in javadoc. Use intervals or List of intervals instead of this.
| // Do not need an interval condition if the interval is ETERNITY | ||
| if (!Intervals.isEternity(interval)) { | ||
| intervals.add(interval); | ||
| boolean intervalsAreEternity = false; |
There was a problem hiding this comment.
Nit:
Suggested change
| boolean intervalsAreEternity = false; | |
| boolean hasEternityInterval = false; |
|
I just pushed up a patch with the review comments. cc @kfaraz |
kfaraz
approved these changes
Nov 25, 2023
|
The error message that comes without this patch is |
cryptoe
added a commit
to cryptoe/druid
that referenced
this pull request
Nov 29, 2023
…ew actions being available on the overlord. (apache#15430) * Fixing failing compaction/parallel index jobs during upgrade due to new actions not available on the overlord. * Fixing build * Removing extra space. * Fixing json getter. * Review comments. (cherry picked from commit a018819)
yashdeep97
pushed a commit
to yashdeep97/druid
that referenced
this pull request
Dec 1, 2023
…ew actions being available on the overlord. (apache#15430) * Fixing failing compaction/parallel index jobs during upgrade due to new actions not available on the overlord. * Fixing build * Removing extra space. * Fixing json getter. * Review comments.
yashdeep97
pushed a commit
to yashdeep97/druid
that referenced
this pull request
Dec 1, 2023
…ew actions being available on the overlord. (apache#15430) * Fixing failing compaction/parallel index jobs during upgrade due to new actions not available on the overlord. * Fixing build * Removing extra space. * Fixing json getter. * Review comments.
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With #15039, a new action got introduced on the overlord
RetrieveSegmentsToReplaceAction. This might not be available on the overlord during upgrade resulting in compaction tasks/reindex tasks failing until the overlord is upgraded.I fixed it by adding an undocumented runtime property
enableConcurrentAppendAndReplacewhich can be set by cluster admins in case they want to try out concurrentAppendAndReplace.This PR also changes
RetrieveSegmentsToReplaceActionto take alist<Intervals>as a input since that is required for MSQ jobs as discussed here : #15284 (comment)