Transform maxclauses fix

[petardz]

Issue #, if available:

Description of changes:

Refactored Transform job to be able to limit amount of modified buckets being processed at a time.
Also added capping of pageSize so that we don't trip that maxClause exception.

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #477 (9c80b29) into main (307ed55) will increase coverage by 0.15%.
The diff coverage is 88.70%.

@@             Coverage Diff              @@
##               main     #477      +/-   ##
============================================
+ Coverage     75.69%   75.85%   +0.15%     
- Complexity     2460     2484      +24     
============================================
  Files           314      315       +1     
  Lines         14456    14502      +46     
  Branches       2236     2244       +8     
============================================
+ Hits          10943    11000      +57     
+ Misses         2253     2248       -5     
+ Partials       1260     1254       -6     

Impacted Files	Coverage Δ
.../org/opensearch/indexmanagement/util/IndexUtils.kt	77.63% <ø> (ø)
...earch/indexmanagement/transform/TransformRunner.kt	78.91% <87.17%> (+0.90%)	⬆️
...ndexmanagement/transform/TransformSearchService.kt	68.33% <87.50%> (+0.51%)	⬆️
...anagement/transform/TransformProcessedBucketLog.kt	93.33% <93.33%> (ø)
...management/indexstatemanagement/MetadataService.kt	62.68% <0.00%> (-2.24%)	⬇️
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	68.71% <0.00%> (-0.59%)	⬇️
...earch/indexmanagement/transform/model/Transform.kt	85.47% <0.00%> (ø)
.../opensearch/indexmanagement/rollup/model/Rollup.kt	86.04% <0.00%> (+0.46%)	⬆️
...ndexstatemanagement/IndexStateManagementHistory.kt	80.00% <0.00%> (+1.37%)	⬆️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[bowenlan-amzn]

Some high level comments:

• Want to confirm is the root cause of this max clause tripping problem this getQueryWithModifiedBuckets populating too many filter queries when there are too many modified buckets?

• Since we are now paginating one execution of continuous transform, do we also need to change this line: +1 on pageProcessed. e.g. Say the page size specified by user is a large number like 10000, it seems we have to cap it to a smaller value like 500 when executing the transform iteration, then page processed will +1 after 20 iterations. What do you think? @petardz

[petardz]

@bowenlan-amzn

Some high level comments:

* Want to confirm is the root cause of this max clause tripping problem this [getQueryWithModifiedBuckets](https://github.com/opensearch-project/index-management/blob/main/src/main/kotlin/org/opensearch/indexmanagement/transform/TransformSearchService.kt#L200) populating too many filter queries when there are too many modified buckets?

Exactly.

* Since we are now paginating one execution of continuous transform, do we also need to change this line: [+1 on pageProcessed](https://github.com/opensearch-project/index-management/blob/main/src/main/kotlin/org/opensearch/indexmanagement/transform/TransformRunner.kt#L203). e.g. Say the page size specified by user is a large number like 10000, it seems we have to cap it to a smaller value like 500 when executing the transform iteration, then page processed will +1 after 20 iterations. What do you think? @petardz

Meaning of pageProcessed was: "number of recomputation(aggregation search) requests executed against source_index" and it still is. The only confusing part could be that user was expecting pageSize=1000 to be used, but pageSize=100 was used. We could add additional field to stats: actualPageSize or pageSizeUsed if necessary.

[bowenlan-amzn]

Beautiful fix. Thanks!

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-477-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7475cfd5d4ef6ba88691de16c5759e6d6bdd6f13
# Push it to GitHub
git push --set-upstream origin backport/backport-477-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-477-to-2.x.