Refactor distribution job for OpenSearch and OpenSearch Dashboards

[tianleh]

The current distribution jobs become more complicated due to support of architecture (x64, arm64), snapshot or regular build, distribution (rpm vs tar). Also there are integ/bwc tests running post build. We have witnessed duplicated logic in these Jenkins files.

https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile
https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch-dashboards/distribution-build.jenkinsfile

This issue is to refactor the common logic into dedicated Groovy files to avoid duplicated code.

Sub tasks

• [META] Move snapshot build jobs out of distribution build jobs #3185
  1. Run a campaign across all repos to move the snapshot creation using GHA along with publishing it to Maven instead of using our Jenkins environment
• Create separate distribution build jobs by supported types (i.e. 1 for TAR, 1 for RPM etc..)
  1. Each job with support both X64 and ARM64 architecture (except for Windows which only support X64)
• Remove hard-coded job name reference in our Python code (if present)
• Update /latest URL re-rewrite logic
• Add support for granular re-build logic in distribution build job
  1. Build only plugins that has been updated and re-use other artifacts as-is from previous builds
  2. Add Bootstrap logic for OSD to support the granular plugin installation feature
• Add necessary tests for all Jenkins files
• De-couple all tests from distribution build jobs
• Link distribution level build id’s with master UUID ( Master job ↔ Child jobs)

Acceptance Criteria

• Each distribution runs independently and has its own integ, bwc and perf tests. Also each tests should run independently and not block the entire build. (Trigger and forget)
• Latest URL logic should be fixed to grab the latest tarball, rpm, windows artifacts individually. Example: /latest/tar, /latest/rpm, etc
• check-for-build should be used to trigger all above workflows and can be used as a orchestrator job build id.

[tianleh]

Notice that OSD has a two stage build and assemble which is different from OS.

cc @peterzhuamazon

[abhinavGupta16]

[triage] This is worth looking into. We can move the common code to a common jenkins library with relevant parameters and call it within the jobs. It would reduce duplicate code and reduce the size and complexity.

Although, this needs to be researched if moving it out in a common library is feasible.

[bbarani]

@peterzhuamazon @tianleh Is this still a valid issue? Can you provide some updates on this issue?

[bbarani]

@tianleh Can you please provide an update?

[gaiksaya]

Approach

We can separate the distribution workflow based on distributions as follow for both OS and OSD separately:

• Tarball: This workflow will do the following:

  • build tarball distribution
  • integ-test
  • bwc-test
  • performance-test

• RPM: This workflow will do the following:

  • build RPM distribution
  • integ-test
  • bwc-test
  • performance-test

• DEB: This workflow will do the following:

  • build deb distribution
  • integ-test
  • bwc-test
  • performance-test

• Docker: This workflow will do the following:

  • build docker distribution
  • integ-test
  • bwc-test
  • performance-test

• Windows: This workflow will do the following:

  • build windows distribution
  • integ-test
  • bwc-test
  • performance-test

• Macos: This workflow will do the following:

  • build macos distribution
  • integ-test
  • bwc-test
  • performance-test

[gaiksaya]

At this point we are not planning on separating the workflows but the individual /latest for each distribution is up.
Closing this issue.
Thanks!