🌱 Add test scripts.

[xuezhaojun]

Summary

There are 2 scripts added:

Sometimes, we want to run integration/e2e test multiple times to reproduce the flaky error, this script allows you to run test many times util it's failed, then it collects then output into _output/flaky-error-test/ directory:

OCM git:(main) ✗ sh hack/integration_flaky_error_test.sh 10
Running test 1 of 2...
Test 1 passed
Running test 2 of 2...
...
Test 7 failed
Test 7 result stored in _output/flaky-error-test/integration_test_20240702-135926_7.output

---

Actually it tests out a integration and a e2e flaky error already:

�[38;5;243m/Users/xuezhao/workspaces/OCM/test/integration/addon/addon_manager_upgrade_test.go:180�[0m

  �[38;5;9m[FAILED] Failed after 3.010s.
  Expected
      <*errors.errorString | 0xc000fbfca0>: 
      unexpected condition matches before duration
      {
          s: "unexpected condition matches before duration",
      }
  to be nil

---

• [FAILED] [360.103 seconds]
Enable addon management feature gate [BeforeEach] Template type addon should be functioning [addon-manager]
  [BeforeEach] /home/runner/work/ocm/ocm/test/e2e/addonmanagement_test.go:68
  [It] /home/runner/work/ocm/ocm/test/e2e/addonmanagement_test.go:171

  [FAILED] Timed out after 300.001s.
  Expected success, but got an error:
      <*errors.errorString | 0xc000a74c20>: 
      the addon e2e-managedcluster-n655ww/hello-template available condition is not true, [{Progressing True 0 2024-07-04 01:45:49 +0000 UTC Progressing progressing... work is not ready} {RegistrationApplied True 0 2024-07-04 01:45:49 +0000 UTC SetPermissionApplied Registration of the addon agent is configured} {ClusterCertificateRotated False 0 2024-07-04 01:45:49 +0000 UTC ClientCertificateUpdateFailed Failed to rotated client certificate namespaces "open-cluster-management-agent-addon" not found} {ManifestApplied False 0 2024-07-04 01:45:49 +0000 UTC AddonManifestAppliedFailed failed to apply the manifests of addon} {Available False 0 2024-07-04 01:45:49 +0000 UTC WorkNotApplied 3 of 4 resources are not available}]
      {
          s: "the addon e2e-managedcluster-n655ww/hello-template available condition is not true, [{Progressing True 0 [202](https://github.com/open-cluster-management-io/ocm/actions/runs/9787073322/job/27022975921?pr=545#step:10:203)4-07-04 01:45:49 +0000 UTC Progressing progressing... work is not ready} {RegistrationApplied True 0 2024-07-04 01:45:49 +0000 UTC SetPermissionApplied Registration of the addon agent is configured} {ClusterCertificateRotated False 0 2024-07-04 01:45:49 +0000 UTC ClientCertificateUpdateFailed Failed to rotated client certificate namespaces \"open-cluster-management-agent-addon\" not found} {ManifestApplied False 0 2024-07-04 01:45:49 +0000 UTC AddonManifestAppliedFailed failed to apply the manifests of addon} {Available False 0 2024-07-04 01:45:49 +0000 UTC WorkNotApplied 3 of 4 resources are not available}]",
      }
  In [BeforeEach] at: /home/runner/work/ocm/ocm/vendor/github.com/onsi/gomega/internal/async_assertion.go:145 @

I will work on this soon.

[xuezhaojun]

/assign @qiujian16

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.41%. Comparing base (5dd2123) to head (31db60a).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #545      +/-   ##
==========================================
+ Coverage   62.39%   62.41%   +0.02%     
==========================================
  Files         138      177      +39     
  Lines       11605    13857    +2252     
==========================================
+ Hits         7241     8649    +1408     
- Misses       3598     4339     +741     
- Partials      766      869     +103     

Flag	Coverage Δ
unit	62.41% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[qiujian16]

/approve
/lgtm

[qiujian16]

But I think the flaky is mainly on e2e?

[xuezhaojun]

Agree, hold the PR to add e2e script as well.
/hold

[xuezhaojun]

/unhold

[xuezhaojun]

@qiujian16 The scripts updated, please take another look

[qiujian16]

/approve
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiujian16, xuezhaojun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [qiujian16]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment