[test] Split CI.yaml

[pingleig]

Background

Currently CI.yaml is too big (the problem is the matrix size is too big) and github workflow does not allow retry single job, you have to retry all jobs in a workflow.

We should be able to split it so it's easier for debugging and retry. The main problem is there is no dependency between workflows https://git.luolix.topmunity/t/how-do-i-specify-job-dependency-running-in-another-workflow/16482/14 However, there are workarounds to check github API for status of other workflow. https://github.com/asarkar/build-time-tracker/blob/03259fad3af0b2a44eb1a451635bb972e3930b52/.github/workflows/publish.yml#L21

TODO

• decide how we want to split the yaml.
• allow trigger CI workflow directly with parameters to filter out test env and test cases using input, e.g. platform=ecs, workload=trace, exporter=oltp
• when generating the matrix, only retry tests that already failed ... (might put it into the util in test framework)
  • might be able to (ab)use the cache feature https://github.com/actions/cache#skipping-steps-based-on-cache-hit
  • Add SSM build and speed up CI re-run #436 and following PRs enabled cache for most workflows
• (optional) use static matrix so we can run action locally (bug) fromJSON leading to a yaml unmarshal error nektos/act#284 (comment) (or we can't help fix act but a lot of change is required in their codebase because they does not evaluate matrix using output from previous jobs).

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stall for 30 days with no activity.