[FEATURE] Run tests in parallel

[tianleh]

Is your feature request related to a problem?
As the number of tests grow in this repo, it takes longer to run all plugin tests. It is ~20mins now
https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/1936082266

For PRs, we need to wait for such time to see the results.

What solution would you like?
The tests can run in parallel.

What alternatives have you considered?

Do you have any additional context?

[tianleh]

@opensearch-project/opensearch-dashboards-test

brainstorming https://docs.cypress.io/guides/guides/parallelization

[kavilla]

After a quick glance it seems like the docs are suggesting run them all multiple machines which is possible but kind of greedy. This seems like a well maintained npm package: https://github.com/tnicola/cypress-parallel that helps it run on the same machine in parallelization. Granted it probably resource intensive for the single machine running it.

[tianleh]

@peterzhuamazon

[peterzhuamazon]

Hi,

Thinking about adding a --cypress-component parameter or similar to control script_finder to call only the corresponding ts files, so that tests can be properly parallelized horizontally, will test later.

The above mentioned cypress parallelization is more of a vertical scale, not suitable for the cases running in Jenkins or local without high spec servers.

Thanks.

[peterzhuamazon]

Opened a build repo issue on opensearch-project/opensearch-build#2144 and will try to use --test parameter as it is matching the test_finder.sh.

[peterzhuamazon]

• 20220524: Add test component selector to run specific tests #234

[peterzhuamazon]

Hi,

I have discovered multiple issues where some test being flaky if run after other tests, some of them will fail as well.

Ex: Alerting more likely to fail if datasource test running beforehand, etc.

The Parallel run will help us with that.

We need to start coding the python code in build repo to consume changes in #234.

Thanks.

[ashwin-pc]

After a quick glance it seems like the docs are suggesting run them all multiple machines which is possible but kind of greedy. This seems like a well maintained npm package: https://github.com/tnicola/cypress-parallel that helps it run on the same machine in parallelization. Granted it probably resource intensive for the single machine running it.

This looks like its no longer maintained with the last update being in Jan. Its also does not have a stable version release yet.

brainstorming https://docs.cypress.io/guides/guides/parallelization

I like this. Yes its asking for multiple VM (Seems like a docker container will do too) but given what Cypress actually does it kinda makes sense. Its also the officially documented process for it. I'd assume Cypress dev's would not have gone this route if they didnt think there was a better alternative.

From their docs:

While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.

They have a neat youtube tutorial on how to setup parallelization for Cypress tests in Github CI: https://www.youtube.com/watch?v=96Yn_IiQUJI. Uses docker containers.