Add BWC tests for running against distribution bundle

[zelinh]

Description

This PR would also include my previous PR #519

Add option to run BWC tests in distribution level, which is running tests in the test cluster with latest distribution bundle installed and all plugins (included in the latest manifest) present.

The command to run BWC tests in distribution level would be ./gradlew bwcTestSuite -Dtests.security.manager=false -PcustomDistributionDownloadType=bundle
If property customDistributionDownloadType is not set nor set to bundle, the BWC tests would be default to run in plugins level as previously configured.

Issues Resolved

Part of opensearch-project/opensearch-build#2870

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #587 (91b241d) into 2.x (41139ab) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                2.x     #587   +/-   ##
=========================================
  Coverage     70.74%   70.74%           
  Complexity      112      112           
=========================================
  Files           124      124           
  Lines          3951     3951           
  Branches        617      630   +13     
=========================================
  Hits           2795     2795           
  Misses          976      976           
  Partials        180      180           

Flag	Coverage Δ
dashboards-notifications	86.40% <ø> (ø)
opensearch-notifications	61.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[qreshi]

I think this was asked in another similar PR but is there a reason we're not submitting this against main first? Our BWC should be running fine there so I'd prefer we go in order.

[zelinh]

I think this was asked in another similar PR but is there a reason we're not submitting this against main first? Our BWC should be running fine there so I'd prefer we go in order.

Currently the distribution build for our 3.0.0 are not stable and I haven't validated these tests for major version update. So we are targeting to 2.5 version for now. I could draft a new PR to backport on main when we getting close to it.

[zelinh]

Confirmed with @qreshi that the CI failure on dashboards notification is a known issue and not related to this PR. Merging now.