fix bug Observability plugin calling createIndex from onNodeStarted m…

[YANG-DB]

Description

BUG - fix bug Observability plugin calling createIndex from onNodeStarted method before cluster state is ready (waits for the cluster state to be ready)

   override fun afterStart() {
           ....
           // wait for the cluster here until it will finish managed node election
            while (clusterService.state().blocks().hasGlobalBlockWithStatus(RestStatus.SERVICE_UNAVAILABLE)) {
                log.info("Wait for cluster to be available ...");
                TimeUnit.SECONDS.sleep(1);
            }
 

To introduce additional start command for all CI build which rely on the OpnSearch Docker image, as Github rollout the deprecation of the Node 16 on all it's CI-runner, as the result all existing Github which rely on the old version of Node.JS (Ex: actions/checkout@v3) failed due to the following errors:

/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

The issue has been discovered over opensearch-project/opensearch-build#5178 and fix has been verified and applied on multiple OpenSearch plugin, such as:

• Enable custom start commands and options to resolve GHA issues ml-commons#3223
• Enable custom start commands and options to resolve GHA issues asynchronous-search#676

Related Issues

• [BUG] Observability plugin calling createIndex from onNodeStarted method in 2.x branches #1883
• Partially resolves: [FEATURE] Get well plan for the Github CI pipeline sql#3168

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ps48]

Thanks for the fix!
I had the same question that was raised by Shwetha in the original issue:

 The main branch doesn't have same code as well? why is there divergence?

Do we know why was this only needed for 2.x and what's the path ahead to converge these branches?

[YANG-DB]

@ps48 I'm not sure why this was removed in the main branch
not really involved recently with this repository....

[peterzhuamazon]

Thanks can be removed as it is deprecated by github.

[peterzhuamazon]

Use v4, as v3 has been deprecated by github.

[peterzhuamazon]

Same as above.

[ps48]

There was breaking change in the @v4 artifact upload workflow which is leading to:

Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

Can you please update the uploaded artifact name to be generated from the matrix used in the workflow. Here's an example of similar issue fixed in the dashboards-observability: https://github.com/opensearch-project/dashboards-observability/pull/2259/files