Fix flaky tests for Bucket-Level Monitor

[qreshi]

Signed-off-by: Mohammad Qureshi 47198598+qreshi@users.noreply.github.com

Issue #, if available: #298

Description of changes:
Monitor configurations are randomized during the plugin integration tests for the components that aren't being explicitly tested for specific scenarios so the test cases aren't overfit for a single situation. For Bucket-Level Monitors, this could lead to the Actions under a Trigger to sometimes be empty (0 Actions) or contain Actions in such a configuration that it no-ops (i.e. PerExecutionScope with no Alert categories configured). These types of configurations were rare but led to resulting Alerts to either be null (for new Alerts) or not updated (for existing Alerts) in some tests which failed assertions, making these tests flaky. This PR updates those tests to account for this scenario to ensure that these cases don't happen.

Commands used to reproduce the flaky tests (confirmed they passed after the changes):

./gradlew ':alerting:integTest' --tests "org.opensearch.alerting.MonitorRunnerIT.test bucket-level monitor with acknowledged alert" -Dtests.seed=39548DC170E2D9BD -Dtests.security.manager=false -Dtests.locale=hr-HR -Dtests.timezone=PNT -Druntime.java=14


./gradlew ':alerting:integTest' --tests "org.opensearch.alerting.MonitorRunnerIT.test bucket-level monitor alert creation and completion" -Dtests.seed=B2CAC72979FAC242 -Dtests.security.manager=false -Dtests.locale=ro-RO -Dtests.timezone=Europe/Lisbon -Druntime.java=14

Re-ran tests multiple times afterwards to check for any other flaky tests:

for i in {1..40}; do ./gradlew ':alerting:integTest' &> /dev/null || echo "Run $i failed"; done

CheckList:
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #318 (1424f72) into main (db0d59e) will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #318      +/-   ##
============================================
+ Coverage     78.48%   78.49%   +0.01%     
- Complexity      216      217       +1     
============================================
  Files           173      173              
  Lines          6975     6975              
  Branches        916      916              
============================================
+ Hits           5474     5475       +1     
+ Misses         1009     1008       -1     
  Partials        492      492              

Impacted Files	Coverage Δ
.../kotlin/org/opensearch/alerting/core/JobSweeper.kt	69.50% <0.00%> (-0.50%)	⬇️
...ain/kotlin/org/opensearch/alerting/AlertService.kt	78.53% <0.00%> (ø)
...ing/destination/client/DestinationEmailClient.java	77.50% <0.00%> (+5.00%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db0d59e...1424f72. Read the comment docs.