[BUG] Flaky Test - search.aggregation/20_terms/string profiler via global ordinals

[penghuo]

Describe the bug

1. Test random failed, Run search.aggregation/20_terms/string profiler via global ordinals 500 times, failed one time.
2. Failure case is not reproducible.

To Reproduce
for i in {1..500}; do ./gradlew ":rest-api-spec:yamlRestTest" --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/20_terms/string profiler via global ordinals}"; done

Expected behavior
Test always successful

Screenshots

REPRODUCE WITH: ./gradlew ':rest-api-spec:yamlRestTest' --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/20_terms/string profiler via global ordinals}" -Dtests.seed=8473F8C0D72BF2FF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ro -Dtests.timezone=Europe/Tiraspol -Druntime.java=17

org.opensearch.test.rest.ClientYamlTestSuiteIT > test {p0=search.aggregation/20_terms/string profiler via global ordinals} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/20_terms:863]: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
    profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
        at __randomizedtesting.SeedInfo.seed([8473F8C0D72BF2FF:C27C71A79D79F07]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:442)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

        Caused by:
        java.lang.AssertionError: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
        profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
            at org.opensearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:115)
            at org.opensearch.test.rest.yaml.section.Assertion.execute(Assertion.java:89)
            at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:431)
            ... 37 more


Suite: Test class org.opensearch.test.rest.ClientYamlTestSuiteIT
  1> [2022-02-18T21:19:22,524][INFO ][o.o.t.r.ClientYamlTestSuiteIT] [test] [p0=search.aggregation/20_terms/string profiler via global ordinals] before test
  1> [2022-02-18T21:19:22,625][INFO ][o.o.t.r.ClientYamlTestSuiteIT] [test] initializing REST clients against [http://[::1]:52792, http://127.0.0.1:52793]
  1> [2022-02-18T21:19:23,103][INFO ][o.o.t.r.ClientYamlTestSuiteIT] [test] initializing client, minimum es version [2.0.0], master version, [2.0.0], hosts [http://[::1]:52792, http://127.0.0.1:52793]
  1> [2022-02-18T21:19:24,708][INFO ][o.o.t.r.ClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "took" : 133,
  1>       "timed_out" : false,
  1>       "_shards" : {
  1>         "total" : 1,
  1>         "successful" : 1,
  1>         "skipped" : 0,
  1>         "failed" : 0
  1>       },
  1>       "hits" : {
  1>         "total" : {
  1>           "value" : 4,
  1>           "relation" : "eq"
  1>         },
  1>         "max_score" : null,
  1>         "hits" : [ ]
  1>       },
  1>       "aggregations" : {
  1>         "str_terms" : {
  1>           "doc_count_error_upper_bound" : 0,
  1>           "sum_other_doc_count" : 0,
  1>           "buckets" : [
  1>             {
  1>               "key" : "sheep",
  1>               "doc_count" : 2,
  1>               "max_number" : {
  1>                 "value" : 3.0
  1>               }
  1>             },
  1>             {
  1>               "key" : "cow",
  1>               "doc_count" : 1,
  1>               "max_number" : {
  1>                 "value" : 1.0
  1>               }
  1>             },
  1>             {
  1>               "key" : "pig",
  1>               "doc_count" : 1,
  1>               "max_number" : {
  1>                 "value" : 1.0
  1>               }
  1>             }
  1>           ]
  1>         }
  1>       },
  1>       "profile" : {
  1>         "shards" : [
  1>           {
  1>             "id" : "[kM11E29STrK3iejrDHEbvA][test_1][0]",
  1>             "searches" : [
  1>               {
  1>                 "query" : [
  1>                   {
  1>                     "type" : "MatchAllDocsQuery",
  1>                     "description" : "*:*",
  1>                     "time_in_nanos" : 2929987,
  1>                     "breakdown" : {
  1>                       "set_min_competitive_score_count" : 0,
  1>                       "match_count" : 0,
  1>                       "shallow_advance_count" : 0,
  1>                       "set_min_competitive_score" : 0,
  1>                       "next_doc" : 2754,
  1>                       "match" : 0,
  1>                       "next_doc_count" : 4,
  1>                       "score_count" : 0,
  1>                       "compute_max_score_count" : 0,
  1>                       "compute_max_score" : 0,
  1>                       "advance" : 2271,
  1>                       "advance_count" : 2,
  1>                       "score" : 0,
  1>                       "build_scorer_count" : 4,
  1>                       "create_weight" : 1566798,
  1>                       "shallow_advance" : 0,
  1>                       "create_weight_count" : 1,
  1>                       "build_scorer" : 1358164
  1>                     }
  1>                   }
  1>                 ],
  1>                 "rewrite_time" : 5686,
  1>                 "collector" : [
  1>                   {
  1>                     "name" : "MultiCollector",
  1>                     "reason" : "search_multi",
  1>                     "time_in_nanos" : 1653840,
  1>                     "children" : [
  1>                       {
  1>                         "name" : "EarlyTerminatingCollector",
  1>                         "reason" : "search_count",
  1>                         "time_in_nanos" : 57051
  1>                       },
  1>                       {
  1>                         "name" : "ProfilingAggregator: [str_terms]",
  1>                         "reason" : "aggregation",
  1>                         "time_in_nanos" : 1211357
  1>                       }
  1>                     ]
  1>                   }
  1>                 ]
  1>               }
  1>             ],
  1>             "aggregations" : [
  1>               {
  1>                 "type" : "GlobalOrdinalsStringTermsAggregator",
  1>                 "description" : "str_terms",
  1>                 "time_in_nanos" : 30187606,
  1>                 "breakdown" : {
  1>                   "reduce" : 0,
  1>                   "post_collection_count" : 1,
  1>                   "build_leaf_collector" : 1076342,
  1>                   "build_aggregation" : 28031582,
  1>                   "build_aggregation_count" : 1,
  1>                   "build_leaf_collector_count" : 2,
  1>                   "post_collection" : 51302,
  1>                   "initialize" : 924900,
  1>                   "initialize_count" : 1,
  1>                   "reduce_count" : 0,
  1>                   "collect" : 103480,
  1>                   "collect_count" : 4
  1>                 },
  1>                 "debug" : {
  1>                   "segments_with_multi_valued_ords" : 1,
  1>                   "collection_strategy" : "dense",
  1>                   "segments_with_single_valued_ords" : 1,
  1>                   "deferred_aggregators" : [
  1>                     "max_number"
  1>                   ],
  1>                   "result_strategy" : "terms",
  1>                   "has_filter" : false
  1>                 },
  1>                 "children" : [
  1>                   {
  1>                     "type" : "MaxAggregator",
  1>                     "description" : "max_number",
  1>                     "time_in_nanos" : 2990449,
  1>                     "breakdown" : {
  1>                       "reduce" : 0,
  1>                       "post_collection_count" : 1,
  1>                       "build_leaf_collector" : 2718379,
  1>                       "build_aggregation" : 130081,
  1>                       "build_aggregation_count" : 1,
  1>                       "build_leaf_collector_count" : 2,
  1>                       "post_collection" : 2259,
  1>                       "initialize" : 3829,
  1>                       "initialize_count" : 1,
  1>                       "reduce_count" : 0,
  1>                       "collect" : 135901,
  1>                       "collect_count" : 4
  1>                     }
  1>                   }
  1>                 ]
  1>               }
  1>             ]
  1>           }
  1>         ]
  1>       }
  1>     }
  1>   }
  1> }]
  1> [2022-02-18T21:19:24,975][INFO ][o.o.t.r.ClientYamlTestSuiteIT] [test] [p0=search.aggregation/20_terms/string profiler via global ordinals] after test
  2> REPRODUCE WITH: ./gradlew ':rest-api-spec:yamlRestTest' --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/20_terms/string profiler via global ordinals}" -Dtests.seed=8473F8C0D72BF2FF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ro -Dtests.timezone=Europe/Tiraspol -Druntime.java=17
  2> java.lang.AssertionError: Failure at [search.aggregation/20_terms:863]: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
    profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
        at __randomizedtesting.SeedInfo.seed([8473F8C0D72BF2FF:C27C71A79D79F07]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:442)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

        Caused by:
        java.lang.AssertionError: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
        profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
            at org.opensearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:115)
            at org.opensearch.test.rest.yaml.section.Assertion.execute(Assertion.java:89)
            at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:431)
            ... 37 more
  2> NOTE: leaving temporary files on disk at: /Users/penghuo/oss/OpenSearch-dev/rest-api-spec/build/testrun/yamlRestTest/temp/org.opensearch.test.rest.ClientYamlTestSuiteIT_8473F8C0D72BF2FF-001
  2> NOTE: test params are: codec=Asserting(Lucene87), sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ro, timezone=Europe/Tiraspol
  2> NOTE: Mac OS X 10.16 x86_64/Eclipse Adoptium 17.0.1 (64-bit)/cpus=12,threads=1,free=336284160,total=536870912
  2> NOTE: All tests run in this JVM: [ClientYamlTestSuiteIT]

Tests with failures:
 - org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals}

1 test completed, 1 failed

=== Standard output of node `node{:rest-api-spec:yamlRestTest-0}` ===

Host/Environment (please complete the following information):

• OS: macOS
• Version: b74d71f

Additional context
Add any other context about the problem here.

[penghuo]

same failure in #2147

[penghuo]

Why segments_with_single_valued_ords=1 and segments_with_multi_valued_ords=1

There are 2 leaf collectors which means there are 2 segments.

• segment 1 has 1 doc. GlobalOrdinalMapping returned
• segment 2 has 3 docs and values.getValueCount() == ordinalMap.getValueCount(), SortedSetDocValues returned.

Possible solution

The purpose of this test is to make sure correct segments collect type is used. so we could add test_1/_forcemerge?max_num_segments=1 in test cases to force merge segments.