[Enhancement] Enable auto-generation of Test Manifest and auto-trigger of integTest early on

[peterzhuamazon]

In the recent 2.16.0 release, we've seen issues where plugin maintainers didn't have enough time to debug or fix test failures within the release window, causing delays.

• Every changes on the OpenSearch related plugins requires a rebuild of Release Candidate (RC)
• Several tests are flaky which requires multiple runs to identify the root cause
• There is merely one week between the generation of 1st RC to the final release date

We want to ensure automatic notifications of both distribution failures and integration test failures are sent to the respective repos before the release process starts, allowing plugin owners to act promptly. To address this, the build team aims to improve automation to help plugin owners identify issues earlier. After reviewing the existing automation we have identified a few parts to improve:

• Only the Input Manifest is auto-generated as of now, and the release manager must manually commit a new Test Manifest
• The initial cronjob PR sent by the automation does not enable integTests during the distribution build for new versions

We propose to make these changes:

• Auto-generate Test Manifest alongside Input Manifest whenever a new version is detected
• Always enable the integTest runs by default when updating cronjob of distribution-build Jenkins workflow (TAR for now)

More to follow,

Thanks.

[peterzhuamazon]

PRs:

• Add TEST_PLATFORM and TEST_DISTRIBUTION param to fine-grain control integ-test triggers #4963
• Enable check-for-build to trigger linux-tar integTest during distribution build cron #4975
• Fix the missing comma in check-for-build #4976
• Allow empty TEST paramsters in distribution build workflows #4983
• Auto generate test manifest alongside input manifest in manifest update #4977

[peterzhuamazon]

We now have both the input manifest and the test manifest created whenever there is a new version.
It will also add tarball as the default integTest that will always trigger after dist build in check-for-build cron.

Thanks!