[BUG] Potential Cluster Slowdown/Lags after merging #13748(#14348) #14338(#14391) in 2.15

[peterzhuamazon]

Describe the bug

During 2.15.0 release process, Infra has generated multiple Release Candidates (RC) for testing before launch date.

Between RC3 and RC4, the only two major changes are: opensearch-project/opensearch-build#4785

• [Backport 2.15] Fix unassigned batch allocation (#13748) (#13748) #14348
• [Backport 2.15] Use remote publication flag to decide which custom objects to upload #14391

Once generated RC4, we have observed potential slow downs in Dashboards related plugin integTests.

• RC3 OSD X64 TAR test: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5950/pipeline
• RC4 OSD X64 TAR test: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5964/pipeline

These failures are not all the same, but observe very similar symptoms, as elements are not loaded in on time. These behaviors seems as if OS is slowing down causing OSD GUI to load much slower as a side effect.

• securityDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/arm64/tar/test-results/5963/integ-test/securityDashboards/with-security/stdout.txt

  1) Home(Get Started) page
       should load Home page properly:
     AssertionError: Timed out retrying after 10000ms: Expected to find content: 'Get started' within the selector: 'h1' but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/security/get_started_spec.js:170:10)

• reportsDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/x64/tar/test-results/5964/integ-test/reportsDashboards/without-security/stdout.txt

  1) Cypress
       "before all" hook for "Visits Reporting homepage":
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `div[data-test-subj="sampleDataSetCardflights"]`, but never found it.

Because this error occurred during a `before all` hook we are skipping the remaining tests in the current suite: `Cypress`
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/reports-dashboards/01-create.spec.js:167:8)

• queryWorkbenchDashboards: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.15.0/7742/linux/x64/tar/test-results/5964/integ-test/queryWorkbenchDashboards/with-security/stdout.txt

  1) Test PPL UI
       "before each" hook for "Confirm results are empty":
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `.euiButton__text[title=PPL]`, but never found it.

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Test PPL UI`
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/query-workbench-dashboards/ui.spec.js:207:8)

We are able to reproduce these error locally as well. After switching from RC4 to RC3 artifacts, these errors are gone and test passes.

Please take a look and let us know the results.
Thanks!

Related component

Other

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

OSD integTests passes.

Additional Details

Plugins
All

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):
All Dist/Arch

Additional context
Add any other context about the problem here.

[getsaurabh02]

@SwethaGuptha @shwetathareja @soosinha Can we please verify and root cause if these changes merged between RC3 and RC4 can potentially slow down OS, thereby causing delay of OSD objects and leading to assertion failures (timeouts) on multiple fronts in the dashboard plugins.

cc: @dblock @Pallavi-AWS @rramachand21

[zelinh]

Reproduced the with-security test failure for securityDashboards locally.

  7 passing (36s)
  1 failing

  1) Home(Get Started) page
       should load Home page properly:
     AssertionError: Timed out retrying after 10000ms: Expected to find content: 'Get started' within the selector: 'h1' but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/security/get_started_spec.js:170:10)

[shwetathareja]

@zelinh : The local repro is with RC4? Can you try if it reproduces with RC3 as well?
Also, are you able to profile OSD to confirm if the extra time spent was in OSD itself or due to delay in OS process bootstrapping.

@SwethaGuptha / @shiv0408 to help profile the 2 PRs merged in 2.15 RC4 which are the suspects:
#14348
#14391

[shwetathareja]

The remote cluster state is disabled by default #14391 made some fix related to it and also the bug fix made by #14348 is for fetching unassigned shards details in batch mode which is disabled by default.

The team can compare bootstrap times for OS process with RC3 and RC4

[shiv0408]

@peterzhuamazon I tried bootstrapping the OpenSearch process from RC3 and RC4 binaries. In fact, the time to bootstrap for RC3 (2.671s) was higher than RC4 (2.491s) on my system.

[peternied]

[Triage - attendees 1 2 3 4 5]
@peterzhuamazon Thanks for creating this issue

[peterzhuamazon]

Hi,

@shwetathareja @shiv0408 We use RC3 and RC4 for the test.
And securityDashboards always succeed on RC3 but failed on RC4.

As you can see this PR the only change between RC3 and RC4 code wise is these two PRs:

• 2.15.0 RC4 Changes opensearch-build#4785

Thanks.

[peterzhuamazon]

We are suspecting there is some side effects of these two PRs causing Dashboards to slow down.
On OpenSearch itself since I dont have enough context on the changes, I do not know if they actually slows down the bootstrap time or not.

I do think this is not just on bootstrap time, but rather when running actually queries where some returns slows down causing dashboards to not load elements quick enough in time for the tests to pass. Thanks.

[shiv0408]

@peterzhuamazon these PRs should not cause any side effects as both of these codes only run when specific settings are enabled, which are disabled by default.
I agree from errors it looks like not just bootstrap, but slowdown in loading UI seems to be the cause. There is something else we are missing which is causing these slowdown, these PRs can't be the culprits.

[soosinha]

Found a commit in opensearch-dashboards-functional-test repo : opensearch-project/opensearch-dashboards-functional-test@9c17a6c
This commit had changed the timeout of tests from 60s to 10s due to which a lot tests started failing.
So the PRs in OpenSearch repo mentioned previously should be good to go ahead with.

[peterzhuamazon]

Found a commit in opensearch-dashboards-functional-test repo : opensearch-project/opensearch-dashboards-functional-test@9c17a6c This commit had changed the timeout of tests from 60s to 10s due to which a lot tests started failing. So the PRs in OpenSearch repo mentioned previously should be good to go ahead with.

Thanks @soosinha for the debug together.

I will soon send a PR to revert this commit.

And proceed with the RC5 builds. Sorry for confusion.

[peterzhuamazon]

Revert now:

• [Backport 2.15] Revert "feat: freeze browser version to 112" opensearch-dashboards-functional-test#1422