[BUG] Tracking issue for known flaky tests

[AWSHurneyt]

Please add more flaky tests to this issue as they're discovered; or notes from investigations into the causes of the flaky tests. Reducing flakiness will increase confidence in our automated testing, and reduce the amount of time needed to merge changes as rerunning cypress tests is a time-consuming process.

List of known flaky tests

1. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor
   1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.

CypressError: Timed out retrying after 10000ms: `cy.select()` failed because it could not find a single `<option>` with value or text matching: `avg_products_base_price`

2. query_level_monitor_spec.js > schedule component displays as intended > for a cron expression schedule
   1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
   2. This flaky behavior is also rarely seen impacting the query_level_monitor_spec.js > schedule component displays as intended > for an interval schedule test.

AssertionError: Timed out retrying after 20000ms: Expected to find element: `[data-test-subj="frequency_field"]`, but never found it.
      at Context.eval (https://search-t-nobody-avjv-e2ealerting-3rb2m37iqureafmtutcxcgypem.eu-west-1.es-staging.amazonaws.com/__cypress/tests?p=cypress/integration/plugins/alerting-dashboards-plugin/query_level_monitor_spec.js:580:10)

3. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor seems to be flaky when executed by the 1.3 functional test repo branch. It has failed with the error below.
   1. This seems to relate to the use of {enter} when using cy.type() here. Cypress fails to execute the {enter} press.
      1. The screenshot below helps illustrate this; the enter key press doesn't execute, so the UI doesn't detect that a Time Field has been defined which prevents the subsequent UI from rendering.
      2. A potential solution may involve changing a .blur() call at the end of .type() calls that use {enter}.

AssertionError: Timed out retrying: Expected to find element: `[data-test-subj="addMetricButton"]`, but never found it.

4. cluster_metrics_monitor_spec.js > can be created > for the Cluster Health API seems to occasionally fail when the API is selected from the dropdown before page finishes loading. This causes the selection to reset which causes the test to fail as it expects to see UI elements for the selection. See image screenshot below

5. monitors_dashboard_spec.js > Displays expected number of alerts occasionally fails with the following error (example). May be related to a delay in alert ingestion.

AssertionError: Timed out retrying after 20000ms: Expected to find element: `:contains(monitors_dashboard_cypress_query_level)`, but never found it. Queried from element: <tr.euiTableRow>

6. composite_level_monitor_spec.js > can be edited > by visual editor occasionally fails with the following error (example). Adding a short wait after this cy.visit call (link) helps to alleviate the flakiness; but a more permanent solution should be implemented.

  1) CompositeLevelMonitor
       can be edited
         by visual editor:
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `[data-test-subj="monitors_list_2"]`, but never found it.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/composite_level_monitor_spec.js:339:10)

[AWSHurneyt]

With the changes made to query_level_monitor_spec.js in PR opensearch-project/opensearch-dashboards-functional-test#622, the second test in the list doesn't appear to be failing periodically anymore. I've scratched it off the list, but we will continue to monitor it.

[AWSHurneyt]

Reopening issue as flaky test 5 still seems to be an issue as seen in PR #767

[AWSHurneyt]

Related issue #762

[wang-chenxi]

Hello, in the description it mentioned bucket-level monitor twice, i wonder if one of them is compositeLevelMonitor? i got both bucket-level and composite-level failed for my PR's E2E test. https://github.com/opensearch-project/alerting-dashboards-plugin/actions/runs/6778149217/job/18423118183?pr=794#step:13:307

[AWSHurneyt]

PR opensearch-project/opensearch-dashboards-functional-test#953 should help address flakiness of test monitors_dashboard_spec.js > Displays expected number of alerts when the tests are executed in the FTR for releases.

[AWSHurneyt]

@wang-chenxi bucket level is mentioned twice in the description because there were a few different flaky observations in that suite. Those bucket level entries have been tracked as flaky since before the release of composite monitors.

[AWSHurneyt]

Related issue opensearch-project/opensearch-dashboards-functional-test#1464

[jowg-amazon]

Adding another known flakey test:

• AcknowledgeAlertsModal > "before all" hook for "Acknowledge button disabled when more than 1 trigger selected"

Error:

1) AcknowledgeAlertsModal
       "before all" hook for "Acknowledge button disabled when more than 1 trigger selected":
     CypressError: `cy.request()` timed out waiting `30000ms` for a response from your server.

CI run link: https://github.com/opensearch-project/alerting-dashboards-plugin/actions/runs/11584512757/job/32251792002